[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:15 dist-test-slave-2gms krb5kdc[23455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:15 dist-test-slave-2gms krb5kdc[23455](info): set up 2 sockets
Jun 19 02:56:15 dist-test-slave-2gms krb5kdc[23455](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 02:56:16.054685 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:16 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:16.222536 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:37473
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:37473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:16.544021 23468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:16.544291 23468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:16.544375 23468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:16.551371 23468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:16.551477 23468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:16.551523 23468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:16.551580 23468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:16.551615 23468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:16.561659 23468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:37473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:37473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:16.562882 23468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:16.564067 23468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:16.576469 23473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:16.576538 23474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:16.576496 23476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:16.577472 23468 server_base.cc:1034] running on GCE node
I20260619 02:56:16.578266 23468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:16.579684 23468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:16.580878 23468 hybrid_clock.cc:648] HybridClock initialized: now 1781837776580837 us; error 40 us; skew 500 ppm
I20260619 02:56:16.581240 23468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:16 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837776, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:16.584295 23468 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:16.585777 23468 webserver.cc:460] Webserver started at http://127.22.230.62:42689/ using document root <none> and password file <none>
I20260619 02:56:16.586495 23468 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:16.586673 23468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:16.586982 23468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:16.589877 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "12dbf6c7b5e540e58be1b5f60beec0d6"
format_stamp: "Formatted at 2026-06-19 02:56:16 on dist-test-slave-2gms"
server_key: "c8b704fa057f136440e4df39937a6246"
server_key_iv: "b47abc0d02a19abd347c6615a160eb02"
server_key_version: "encryptionkey@0"
I20260619 02:56:16.590723 23468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "12dbf6c7b5e540e58be1b5f60beec0d6"
format_stamp: "Formatted at 2026-06-19 02:56:16 on dist-test-slave-2gms"
server_key: "c8b704fa057f136440e4df39937a6246"
server_key_iv: "b47abc0d02a19abd347c6615a160eb02"
server_key_version: "encryptionkey@0"
I20260619 02:56:16.595660 23468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 02:56:16.599463 23483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:16.600687 23468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 02:56:16.600857 23468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "12dbf6c7b5e540e58be1b5f60beec0d6"
format_stamp: "Formatted at 2026-06-19 02:56:16 on dist-test-slave-2gms"
server_key: "c8b704fa057f136440e4df39937a6246"
server_key_iv: "b47abc0d02a19abd347c6615a160eb02"
server_key_version: "encryptionkey@0"
I20260619 02:56:16.601051 23468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:16.633910 23468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:16.637728 23468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:16.637995 23468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:16.657441 23468 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:37473
I20260619 02:56:16.657577 23534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:37473 every 8 connection(s)
I20260619 02:56:16.658798 23468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:16.662317 23535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:16.668309 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 23468
I20260619 02:56:16.668586 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:16.670362 23448 external_mini_cluster.cc:1428] Setting key e29d2ed02f55394e6acef513b950486c
I20260619 02:56:16.672529 23535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6: Bootstrap starting.
I20260619 02:56:16.676460 23535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:16.677358 23535 log.cc:824] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:16.680184 23535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6: No bootstrap required, opened a new log
I20260619 02:56:16.688841 23535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37473 } }
I20260619 02:56:16.689445 23535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:16.689563 23535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12dbf6c7b5e540e58be1b5f60beec0d6, State: Initialized, Role: FOLLOWER
I20260619 02:56:16.690199 23535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37473 } }
I20260619 02:56:16.690438 23535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:16.690675 23535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:16.690851 23535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:16.692955 23535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37473 } }
I20260619 02:56:16.693446 23535 leader_election.cc:302] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12dbf6c7b5e540e58be1b5f60beec0d6; no voters:
I20260619 02:56:16.694056 23535 leader_election.cc:288] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:16.694149 23540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:16.695524 23540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [term 1 LEADER]: Becoming Leader. State: Replica: 12dbf6c7b5e540e58be1b5f60beec0d6, State: Running, Role: LEADER
I20260619 02:56:16.696100 23540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37473 } }
I20260619 02:56:16.696532 23535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 02:56:16 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:16.699625 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37473 } } }
I20260619 02:56:16.699994 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:16.704746 23542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12dbf6c7b5e540e58be1b5f60beec0d6. Latest consensus state: current_term: 1 leader_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbf6c7b5e540e58be1b5f60beec0d6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37473 } } }
I20260619 02:56:16.705449 23542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:16.715543 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:16.674008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.22.230.62:37473)
0619 02:56:16.674801 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:16.674811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:16.676084 (+ 1273us) server_negotiation.cc:407] Connection header received
0619 02:56:16.684932 (+ 8848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:16.684950 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:16.685288 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:16.685696 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:16.688423 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:16.690470 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:16.692216 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:16.692873 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:16.693072 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:16.700020 (+ 6948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:16.700034 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:16.700053 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:16.700105 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:16.705831 (+ 5726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:16.707265 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:16.707277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:16.707288 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:16.707579 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:16.708257 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:16.708265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:16.708270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:16.709184 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:16.709542 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:16.712772 (+ 3230us) server_negotiation.cc:299] Negotiation successful
0619 02:56:16.713281 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":378,"threads_started":1}
I20260619 02:56:16.719790 23547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:16.725047 23547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:16.734797 23547 catalog_manager.cc:1347] Generated new cluster ID: fa771919f2484adcbda580cdd470df30
I20260619 02:56:16.734930 23547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:16.772930 23547 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:16.774221 23547 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:16.783304 23547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12dbf6c7b5e540e58be1b5f60beec0d6: Generated new TSK 0
I20260619 02:56:16.784091 23547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:16.836133 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37473
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:17.186532 23561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:17.186872 23561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:17.186971 23561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:17.193903 23561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:17.194001 23561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:17.194119 23561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:17.203547 23561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:17.204810 23561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:17.206194 23561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:17.219157 23567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:17.219241 23566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:17.219216 23569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:17.220206 23561 server_base.cc:1034] running on GCE node
I20260619 02:56:17.220760 23561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:17.221643 23561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:17.222864 23561 hybrid_clock.cc:648] HybridClock initialized: now 1781837777222815 us; error 48 us; skew 500 ppm
I20260619 02:56:17.223162 23561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:17 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837777, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:17.226661 23561 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:17.228053 23561 webserver.cc:460] Webserver started at http://127.22.230.1:39073/ using document root <none> and password file <none>
I20260619 02:56:17.228722 23561 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:17.228858 23561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:17.229168 23561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:17.232012 23561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "713fa11f91ec47f0ae5f9b9706ea73cf"
format_stamp: "Formatted at 2026-06-19 02:56:17 on dist-test-slave-2gms"
server_key: "aa70fdf78b64099d96c0d6fe8b17c3a5"
server_key_iv: "d94e0f3162dc389aabb970478c5366f6"
server_key_version: "encryptionkey@0"
I20260619 02:56:17.232738 23561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "713fa11f91ec47f0ae5f9b9706ea73cf"
format_stamp: "Formatted at 2026-06-19 02:56:17 on dist-test-slave-2gms"
server_key: "aa70fdf78b64099d96c0d6fe8b17c3a5"
server_key_iv: "d94e0f3162dc389aabb970478c5366f6"
server_key_version: "encryptionkey@0"
I20260619 02:56:17.237576 23561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:56:17.240954 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:17.242221 23561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:56:17.242400 23561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "713fa11f91ec47f0ae5f9b9706ea73cf"
format_stamp: "Formatted at 2026-06-19 02:56:17 on dist-test-slave-2gms"
server_key: "aa70fdf78b64099d96c0d6fe8b17c3a5"
server_key_iv: "d94e0f3162dc389aabb970478c5366f6"
server_key_version: "encryptionkey@0"
I20260619 02:56:17.242651 23561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:17.262785 23561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:17.267042 23561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:17.267333 23561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:17.268293 23561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:17.269647 23561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:17.269748 23561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:17.269915 23561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:17.269985 23561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:17.306692 23561 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:36641
I20260619 02:56:17.306735 23688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:36641 every 8 connection(s)
I20260619 02:56:17.307986 23561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:17.313259 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 23561
I20260619 02:56:17.313496 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:17.315441 23448 external_mini_cluster.cc:1428] Setting key 805ad7dda14e23b7bceafcd4a13de98f
Jun 19 02:56:17 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837777, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:17.328712 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:17.310709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:47865 (local address 127.22.230.62:37473)
0619 02:56:17.311120 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:17.311127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:17.312003 (+ 876us) server_negotiation.cc:407] Connection header received
0619 02:56:17.314721 (+ 2718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:17.314727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:17.314832 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:17.314937 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:17.316616 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.317519 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.319178 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.319625 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.319770 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:17.323149 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:17.323172 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:17.323176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:17.323205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:17.325829 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:17.326500 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:17.326512 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:17.326515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:17.326674 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:17.327073 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:17.327078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:17.327081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:17.327356 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:17.327503 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:17.328148 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 02:56:17.328365 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":199,"threads_started":1}
I20260619 02:56:17.329573 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:17.311185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:37473 (local address 127.22.230.1:47865)
0619 02:56:17.311841 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:17.311890 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:17.314276 (+ 2386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:17.315213 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:17.315225 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:17.315781 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:17.316454 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:17.316473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.317685 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.317694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:17.319034 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:17.319045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.319854 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.319862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:17.320097 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:17.320758 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:17.320782 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:17.322936 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:17.325977 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:17.325986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:17.326001 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:17.326379 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:17.326773 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:17.326779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:17.326783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:17.326960 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:17.327479 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:17.327486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:17.327789 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 02:56:17.328050 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 02:56:17.328533 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:56:17.331017 23689 heartbeater.cc:347] Connected to a master server at 127.22.230.62:37473
I20260619 02:56:17.331382 23689 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:17.332187 23689 heartbeater.cc:511] Master 127.22.230.62:37473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:56:17.334489 23500 ts_manager.cc:194] Registered new tserver with Master: 713fa11f91ec47f0ae5f9b9706ea73cf (127.22.230.1:36641)
I20260619 02:56:17.336508 23500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:47865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:17.353225 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37473
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:17.679489 23695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:17.679741 23695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:17.679908 23695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:17.686951 23695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:17.687070 23695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:17.687273 23695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:17.697484 23695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:17.698770 23695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:17.700071 23695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:17.713707 23701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:17.713600 23700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:17.713891 23703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:17.714684 23695 server_base.cc:1034] running on GCE node
I20260619 02:56:17.715240 23695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:17.716076 23695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:17.717269 23695 hybrid_clock.cc:648] HybridClock initialized: now 1781837777717229 us; error 33 us; skew 500 ppm
I20260619 02:56:17.717594 23695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:17 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837777, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:17.720757 23695 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:17.722142 23695 webserver.cc:460] Webserver started at http://127.22.230.2:37065/ using document root <none> and password file <none>
I20260619 02:56:17.722900 23695 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:17.723039 23695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:17.723335 23695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:17.726071 23695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "35cb7a39246e48859f68ce1cc2f8c106"
format_stamp: "Formatted at 2026-06-19 02:56:17 on dist-test-slave-2gms"
server_key: "fd7578a626381771fe11bf3cdac0de23"
server_key_iv: "37a203fede5268682455d24401baf13d"
server_key_version: "encryptionkey@0"
I20260619 02:56:17.726850 23695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "35cb7a39246e48859f68ce1cc2f8c106"
format_stamp: "Formatted at 2026-06-19 02:56:17 on dist-test-slave-2gms"
server_key: "fd7578a626381771fe11bf3cdac0de23"
server_key_iv: "37a203fede5268682455d24401baf13d"
server_key_version: "encryptionkey@0"
I20260619 02:56:17.731889 23695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 02:56:17.735481 23710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:17.736721 23695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:56:17.736886 23695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "35cb7a39246e48859f68ce1cc2f8c106"
format_stamp: "Formatted at 2026-06-19 02:56:17 on dist-test-slave-2gms"
server_key: "fd7578a626381771fe11bf3cdac0de23"
server_key_iv: "37a203fede5268682455d24401baf13d"
server_key_version: "encryptionkey@0"
I20260619 02:56:17.737067 23695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:17.752179 23695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:17.756213 23695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:17.756486 23695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:17.757404 23695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:17.759127 23695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:17.759196 23695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:17.759294 23695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:17.759336 23695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:17.795683 23695 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:34717
I20260619 02:56:17.795742 23822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:34717 every 8 connection(s)
I20260619 02:56:17.796947 23695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:17.799067 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 23695
I20260619 02:56:17.799293 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:17.799788 23448 external_mini_cluster.cc:1428] Setting key d75f528c0c123d5bd43b9516f0eaf409
Jun 19 02:56:17 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837777, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:17.820070 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:17.799574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:55473 (local address 127.22.230.62:37473)
0619 02:56:17.799773 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:17.799779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:17.801258 (+ 1479us) server_negotiation.cc:407] Connection header received
0619 02:56:17.803216 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:17.803229 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:17.803372 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:17.803548 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:17.805010 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.807168 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.808821 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.809998 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.810130 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:17.813949 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:17.813954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:17.813957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:17.813986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:17.816989 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:17.817741 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:17.817747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:17.817753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:17.817901 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:17.818353 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:17.818360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:17.818363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:17.818715 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:17.818861 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:17.819517 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 02:56:17.819737 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:56:17.821060 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:17.800149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:37473 (local address 127.22.230.2:55473)
0619 02:56:17.800978 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:17.801029 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:17.802916 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:17.803754 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:17.803762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:17.804228 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:17.804803 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:17.804820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.807432 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.807436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:17.808650 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:17.808661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:17.810201 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:17.810206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:17.810411 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:17.811113 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:17.811133 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:17.813737 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:17.817149 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:17.817158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:17.817173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:17.817618 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:17.818009 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:17.818015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:17.818018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:17.818245 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:17.818850 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:17.818858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:17.819184 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 02:56:17.819445 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 02:56:17.819946 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":343,"threads_started":1}
I20260619 02:56:17.822515 23823 heartbeater.cc:347] Connected to a master server at 127.22.230.62:37473
I20260619 02:56:17.822878 23823 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
I20260619 02:56:17.823669 23823 heartbeater.cc:511] Master 127.22.230.62:37473 requested a full tablet report, sending...
I20260619 02:56:17.825233 23500 ts_manager.cc:194] Registered new tserver with Master: 35cb7a39246e48859f68ce1cc2f8c106 (127.22.230.2:34717)
I20260619 02:56:17.826021 23500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:55473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:17.845095 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37473
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:18.175853 23828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:18.176105 23828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:18.176206 23828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:18.183425 23828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:18.183521 23828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:18.183645 23828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:18.192925 23828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:18.194072 23828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:18.195308 23828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:18.209038 23833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:18.208986 23834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:18.209064 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:18.210145 23828 server_base.cc:1034] running on GCE node
I20260619 02:56:18.210772 23828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:18.211628 23828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:18.212843 23828 hybrid_clock.cc:648] HybridClock initialized: now 1781837778212803 us; error 55 us; skew 500 ppm
I20260619 02:56:18.213140 23828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:18.216225 23828 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:18.217468 23828 webserver.cc:460] Webserver started at http://127.22.230.3:43479/ using document root <none> and password file <none>
I20260619 02:56:18.218137 23828 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:18.218267 23828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:18.218641 23828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:18.221441 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "8f1fb9702e8345eabfb2182ecd5bdb95"
format_stamp: "Formatted at 2026-06-19 02:56:18 on dist-test-slave-2gms"
server_key: "8ce7028186463fe99ba9b0400a6141e9"
server_key_iv: "40f76e4a95055eae0909fe93479983de"
server_key_version: "encryptionkey@0"
I20260619 02:56:18.222167 23828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "8f1fb9702e8345eabfb2182ecd5bdb95"
format_stamp: "Formatted at 2026-06-19 02:56:18 on dist-test-slave-2gms"
server_key: "8ce7028186463fe99ba9b0400a6141e9"
server_key_iv: "40f76e4a95055eae0909fe93479983de"
server_key_version: "encryptionkey@0"
I20260619 02:56:18.227126 23828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:56:18.230854 23843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:18.232232 23828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:56:18.232445 23828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "8f1fb9702e8345eabfb2182ecd5bdb95"
format_stamp: "Formatted at 2026-06-19 02:56:18 on dist-test-slave-2gms"
server_key: "8ce7028186463fe99ba9b0400a6141e9"
server_key_iv: "40f76e4a95055eae0909fe93479983de"
server_key_version: "encryptionkey@0"
I20260619 02:56:18.232678 23828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:18.263309 23828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:18.267609 23828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:18.267907 23828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:18.268826 23828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:18.270239 23828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:18.270318 23828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:18.270443 23828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:18.270524 23828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:18.307513 23828 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:44709
I20260619 02:56:18.307530 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:44709 every 8 connection(s)
I20260619 02:56:18.309093 23828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:18.311391 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 23828
I20260619 02:56:18.311599 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:18.312014 23448 external_mini_cluster.cc:1428] Setting key a6cd28abac6c15c3b1839a6a204b6bc3
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:18.328980 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.312765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:37215 (local address 127.22.230.62:37473)
0619 02:56:18.312938 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:18.312944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:18.313213 (+ 269us) server_negotiation.cc:407] Connection header received
0619 02:56:18.315706 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:18.315710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:18.315772 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:18.315850 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:18.317234 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.318424 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.320037 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.320433 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.320534 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.323677 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:18.323722 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:18.323729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:18.323777 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:18.326041 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.326794 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.326820 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.326824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.326974 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.327564 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.327570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.327572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.327853 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:18.328027 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:18.328613 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 02:56:18.328760 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:56:18.329865 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.312236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:37473 (local address 127.22.230.3:37215)
0619 02:56:18.313060 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:18.313112 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:18.315412 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:18.316061 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:18.316070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:18.316560 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:18.317081 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:18.317098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.318594 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.318605 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:18.319916 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:18.319926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.320609 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.320613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:18.320791 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.321376 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:18.321396 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:18.323489 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:18.326180 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:18.326185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:18.326197 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:18.326653 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:18.327141 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:18.327163 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:18.327167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:18.327447 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:18.327995 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:18.328000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:18.328294 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 02:56:18.328504 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 02:56:18.328829 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":323,"threads_started":1}
I20260619 02:56:18.331210 23956 heartbeater.cc:347] Connected to a master server at 127.22.230.62:37473
I20260619 02:56:18.331475 23956 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:18.332132 23956 heartbeater.cc:511] Master 127.22.230.62:37473 requested a full tablet report, sending...
I20260619 02:56:18.333389 23500 ts_manager.cc:194] Registered new tserver with Master: 8f1fb9702e8345eabfb2182ecd5bdb95 (127.22.230.3:44709)
I20260619 02:56:18.334198 23500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:37215
I20260619 02:56:18.339699 23689 heartbeater.cc:503] Master 127.22.230.62:37473 was elected leader, sending a full tablet report...
I20260619 02:56:18.340161 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:18.382864 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.368980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.22.230.1:36641)
0619 02:56:18.369618 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:18.369625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:18.369650 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:56:18.369710 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:18.369716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:18.369917 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:18.370056 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:18.371436 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.372470 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.373624 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.374127 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.374232 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.377885 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:18.377895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:18.377909 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:18.377940 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:18.380566 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.381108 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.381114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.381116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.381230 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.381533 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.381554 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.381556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.381844 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:18.382049 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:18.382284 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 02:56:18.382453 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":342,"threads_started":1}
W20260619 02:56:18.384038 23604 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40080
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:18.430610 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.417846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.22.230.1:36641)
0619 02:56:18.418087 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:18.418093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:18.418390 (+ 297us) server_negotiation.cc:407] Connection header received
0619 02:56:18.418530 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:18.418534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:18.418699 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:18.418779 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:18.420046 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.420703 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.421855 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.422245 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.422343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.426049 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:18.426053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:18.426056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:18.426085 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:18.428510 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.429022 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.429025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.429027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.429127 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.429436 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.429439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.429441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.429675 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:18.429842 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:18.430084 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 02:56:18.430240 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:56:18.432001 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 23561
I20260619 02:56:18.443945 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.433026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.22.230.1:36641)
0619 02:56:18.433222 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:18.433227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:18.433301 (+ 74us) server_negotiation.cc:407] Connection header received
0619 02:56:18.433412 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:18.433416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:18.433463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:18.433533 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:18.434543 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.435233 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.436481 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.436874 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.436954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.439285 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:18.439289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:18.439291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:18.439315 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:18.441940 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.442460 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.442464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.442466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.442619 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.442923 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.442926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.442927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.443118 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:18.443295 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:18.443490 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 02:56:18.443615 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:56:18.444628 23684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:18.525969 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 23561
I20260619 02:56:18.542490 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 23695
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:18.556667 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.543334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.22.230.2:34717)
0619 02:56:18.543857 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:18.543864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:18.543888 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:56:18.543945 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:18.543951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:18.544150 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:18.544275 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:18.545342 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.546265 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.547539 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.548047 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.548140 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.551949 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:18.551959 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:18.551972 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:18.552000 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:18.554502 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.555098 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.555104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.555106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.555213 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.555522 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.555524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.555526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.555804 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:18.556013 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:18.556234 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 02:56:18.556405 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":271,"threads_started":1}
I20260619 02:56:18.557837 23818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:18.639504 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 23695
I20260619 02:56:18.661345 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 23828
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:18.676986 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:18.662404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.22.230.3:44709)
0619 02:56:18.663017 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:18.663023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:18.663049 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:56:18.663104 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:18.663110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:18.663329 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:18.663501 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:18.664689 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.665669 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.667119 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:18.667682 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:18.667819 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:18.671780 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:18.671790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:18.671809 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:18.671839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:18.674283 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.675018 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.675033 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.675035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.675162 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:18.675541 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:18.675550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:18.675552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:18.675845 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:18.676142 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:18.676422 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:56:18.676665 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":325,"threads_started":1}
I20260619 02:56:18.678188 23951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:18.757902 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 23828
I20260619 02:56:18.774132 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 23468
I20260619 02:56:18.775264 23530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:18.875458 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 23468
2026-06-19T02:56:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4001 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23977](info): set up 2 sockets
Jun 19 02:56:18 dist-test-slave-2gms krb5kdc[23977](info): commencing operation
krb5kdc: starting...
W20260619 02:56:19.942685 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:20 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:20.120045 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:46649
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:46649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:20.424337 23990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:20.424754 23990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:20.424902 23990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:20.432085 23990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:20.432230 23990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:20.432333 23990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:20.432430 23990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:20.432492 23990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:20.442365 23990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:46649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:20.443624 23990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:20.444890 23990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:20.456323 23995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:20.456405 23998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:20.456372 23996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:20.457160 23990 server_base.cc:1034] running on GCE node
I20260619 02:56:20.457840 23990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:20.459187 23990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:20.460379 23990 hybrid_clock.cc:648] HybridClock initialized: now 1781837780460327 us; error 47 us; skew 500 ppm
I20260619 02:56:20.460706 23990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:20 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837780, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:20.468394 23990 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:20.469691 23990 webserver.cc:460] Webserver started at http://127.22.230.62:41473/ using document root <none> and password file <none>
I20260619 02:56:20.470319 23990 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:20.470413 23990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:20.470768 23990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:20.473527 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "dfabbb75811f486082c817a906dea9fb"
format_stamp: "Formatted at 2026-06-19 02:56:20 on dist-test-slave-2gms"
server_key: "986bdbb40de74f8fbe2c95e957c3b0fa"
server_key_iv: "81720b0bee958f0bae84a01d8bd75165"
server_key_version: "encryptionkey@0"
I20260619 02:56:20.474189 23990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "dfabbb75811f486082c817a906dea9fb"
format_stamp: "Formatted at 2026-06-19 02:56:20 on dist-test-slave-2gms"
server_key: "986bdbb40de74f8fbe2c95e957c3b0fa"
server_key_iv: "81720b0bee958f0bae84a01d8bd75165"
server_key_version: "encryptionkey@0"
I20260619 02:56:20.479100 23990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:56:20.482158 24005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:20.483492 23990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:20.483665 23990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "dfabbb75811f486082c817a906dea9fb"
format_stamp: "Formatted at 2026-06-19 02:56:20 on dist-test-slave-2gms"
server_key: "986bdbb40de74f8fbe2c95e957c3b0fa"
server_key_iv: "81720b0bee958f0bae84a01d8bd75165"
server_key_version: "encryptionkey@0"
I20260619 02:56:20.483850 23990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:20.497365 23990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:20.501442 23990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:20.501724 23990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:20.519412 23990 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:46649
I20260619 02:56:20.519431 24056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:46649 every 8 connection(s)
I20260619 02:56:20.520714 23990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:20.524158 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 23990
I20260619 02:56:20.524389 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:20.524437 24057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:20.525022 23448 external_mini_cluster.cc:1428] Setting key b241f19e27cd65a59406bfc37de99ad0
I20260619 02:56:20.537489 24057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb: Bootstrap starting.
Jun 19 02:56:20 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:20.540892 24057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:20.542062 24057 log.cc:824] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:20.544895 24057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb: No bootstrap required, opened a new log
I20260619 02:56:20.547842 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:20.526934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.22.230.62:46649)
0619 02:56:20.527741 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:20.527751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:20.527792 (+ 41us) server_negotiation.cc:407] Connection header received
0619 02:56:20.529937 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:20.529961 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:20.530355 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:20.530734 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:20.532338 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:20.533541 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:20.535359 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:20.535969 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:20.536091 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:20.540485 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:20.540492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:20.540504 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:20.540540 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:20.543945 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:20.544583 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:20.544588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:20.544595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:20.544740 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:20.545158 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:20.545163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:20.545165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:20.545644 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:20.545943 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:20.546437 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 02:56:20.546893 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":384,"threads_started":1}
I20260619 02:56:20.552904 24057 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfabbb75811f486082c817a906dea9fb" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46649 } }
I20260619 02:56:20.553359 24057 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:20.553495 24057 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfabbb75811f486082c817a906dea9fb, State: Initialized, Role: FOLLOWER
I20260619 02:56:20.554111 24057 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfabbb75811f486082c817a906dea9fb" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46649 } }
I20260619 02:56:20.554342 24057 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:20.554455 24057 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:20.554643 24057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:20.556674 24057 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfabbb75811f486082c817a906dea9fb" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46649 } }
I20260619 02:56:20.557132 24057 leader_election.cc:302] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfabbb75811f486082c817a906dea9fb; no voters:
I20260619 02:56:20.557689 24057 leader_election.cc:288] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:20.557785 24062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:20.559239 24062 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [term 1 LEADER]: Becoming Leader. State: Replica: dfabbb75811f486082c817a906dea9fb, State: Running, Role: LEADER
I20260619 02:56:20.559706 24057 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:20.559698 24062 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfabbb75811f486082c817a906dea9fb" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46649 } }
I20260619 02:56:20.562434 24063 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfabbb75811f486082c817a906dea9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfabbb75811f486082c817a906dea9fb" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46649 } } }
I20260619 02:56:20.562407 24064 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfabbb75811f486082c817a906dea9fb. Latest consensus state: current_term: 1 leader_uuid: "dfabbb75811f486082c817a906dea9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfabbb75811f486082c817a906dea9fb" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46649 } } }
I20260619 02:56:20.562915 24064 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:20.562915 24063 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:20.563817 24069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:20.569455 24069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:20.577678 24069 catalog_manager.cc:1347] Generated new cluster ID: af58fe185e6d44e792260fdcd368ea73
I20260619 02:56:20.577788 24069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:20.603751 24069 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:20.605067 24069 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:20.614405 24069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfabbb75811f486082c817a906dea9fb: Generated new TSK 0
I20260619 02:56:20.615195 24069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:20.658694 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46649
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:20.971099 24083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:20.971370 24083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:20.971463 24083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:20.978765 24083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:20.978869 24083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:20.978981 24083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:20.988547 24083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:20.989660 24083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:20.990906 24083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:21.003669 24088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:21.003752 24089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:21.003778 24091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:21.004779 24083 server_base.cc:1034] running on GCE node
I20260619 02:56:21.005336 24083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:21.006278 24083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:21.007539 24083 hybrid_clock.cc:648] HybridClock initialized: now 1781837781007480 us; error 55 us; skew 500 ppm
I20260619 02:56:21.007853 24083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:21 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:21.010927 24083 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:21.012256 24083 webserver.cc:460] Webserver started at http://127.22.230.1:42655/ using document root <none> and password file <none>
I20260619 02:56:21.012936 24083 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:21.013070 24083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:21.013372 24083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:21.016184 24083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "7642baa8b69140c5857ce685743844a6"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "14a0a4997c86a55d4f8eee0d1d25184b"
server_key_iv: "90ded595e53f6f42ad121ac382837737"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.016927 24083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "7642baa8b69140c5857ce685743844a6"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "14a0a4997c86a55d4f8eee0d1d25184b"
server_key_iv: "90ded595e53f6f42ad121ac382837737"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.022197 24083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 02:56:21.025455 24098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.026824 24083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:21.026995 24083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "7642baa8b69140c5857ce685743844a6"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "14a0a4997c86a55d4f8eee0d1d25184b"
server_key_iv: "90ded595e53f6f42ad121ac382837737"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.027227 24083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:21.040787 24083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:21.044872 24083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:21.045156 24083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:21.046056 24083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:21.047398 24083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:21.047467 24083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.047636 24083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:21.047703 24083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.079862 24083 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:46727
I20260619 02:56:21.079936 24210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:46727 every 8 connection(s)
I20260619 02:56:21.081110 24083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:21.083999 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24083
I20260619 02:56:21.084227 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:21.084821 23448 external_mini_cluster.cc:1428] Setting key 3e8a8eb356ac8f7765a4c427370f3261
Jun 19 02:56:21 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:56:21.100751 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:21.083681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:53243 (local address 127.22.230.62:46649)
0619 02:56:21.084138 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:21.084144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:21.085128 (+ 984us) server_negotiation.cc:407] Connection header received
0619 02:56:21.087275 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:21.087280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:21.087359 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:21.087480 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:21.088999 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.089722 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.091301 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.091694 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.091824 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:21.095345 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:21.095351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:21.095354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:21.095383 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:21.097894 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:21.098604 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:21.098611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:21.098615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:21.098732 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:21.099133 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:21.099137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:21.099140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:21.099398 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:21.099548 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:21.100171 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 02:56:21.100379 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":227,"threads_started":1}
I20260619 02:56:21.101663 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:21.084138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46649 (local address 127.22.230.1:53243)
0619 02:56:21.084956 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:21.085005 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:21.087013 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:21.087718 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:21.087726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:21.088298 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:21.088841 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:21.088857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.089882 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.089886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:21.091163 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:21.091174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.091901 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.091910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:21.092159 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:21.092824 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:21.092843 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:21.095115 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:21.098029 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:21.098038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:21.098052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:21.098444 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:21.098840 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:21.098845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:21.098849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:21.099020 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:21.099521 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:21.099528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:21.099842 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 02:56:21.100083 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 02:56:21.100573 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:56:21.103194 24211 heartbeater.cc:347] Connected to a master server at 127.22.230.62:46649
I20260619 02:56:21.103500 24211 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:21.104182 24211 heartbeater.cc:511] Master 127.22.230.62:46649 requested a full tablet report, sending...
I20260619 02:56:21.106415 24022 ts_manager.cc:194] Registered new tserver with Master: 7642baa8b69140c5857ce685743844a6 (127.22.230.1:46727)
I20260619 02:56:21.107882 24022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:53243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:21.123026 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46649
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:21.433565 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:21.433884 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:21.434046 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:21.441318 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:21.441463 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:21.441629 24217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:21.451152 24217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:21.452397 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:21.453677 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:21.466321 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:21.466383 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:21.466418 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:21.467157 24217 server_base.cc:1034] running on GCE node
I20260619 02:56:21.467690 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:21.468535 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:21.469754 24217 hybrid_clock.cc:648] HybridClock initialized: now 1781837781469715 us; error 49 us; skew 500 ppm
I20260619 02:56:21.470089 24217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:21 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:21.473063 24217 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:21.474382 24217 webserver.cc:460] Webserver started at http://127.22.230.2:36197/ using document root <none> and password file <none>
I20260619 02:56:21.475051 24217 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:21.475147 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:21.475394 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:21.478127 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "ca547841fb7c435685e8859805a530e7"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "148cc18fe43a7ce7da23189573e92368"
server_key_iv: "5a5b89b0ae0bf764214bc6965609cef7"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.478890 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "ca547841fb7c435685e8859805a530e7"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "148cc18fe43a7ce7da23189573e92368"
server_key_iv: "5a5b89b0ae0bf764214bc6965609cef7"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.483592 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:56:21.486713 24232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.487998 24217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:56:21.488190 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "ca547841fb7c435685e8859805a530e7"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "148cc18fe43a7ce7da23189573e92368"
server_key_iv: "5a5b89b0ae0bf764214bc6965609cef7"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.488384 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:21.508677 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:21.512284 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:21.512576 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:21.513557 24217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:21.514855 24217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:21.514967 24217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.515074 24217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:21.515120 24217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.549117 24217 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:39823
I20260619 02:56:21.549218 24344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:39823 every 8 connection(s)
I20260619 02:56:21.550369 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:21.558354 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24217
I20260619 02:56:21.558552 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:21.559048 23448 external_mini_cluster.cc:1428] Setting key 3ea6eba5ce1056cdf00932bf59c30942
Jun 19 02:56:21 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:21.570102 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:21.552819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:38121 (local address 127.22.230.62:46649)
0619 02:56:21.552996 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:21.553000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:21.554121 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 02:56:21.555977 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:21.555982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:21.556061 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:21.556178 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:21.557595 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.558288 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.560232 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.560771 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.560873 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:21.564227 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:21.564237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:21.564244 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:21.564286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:21.567041 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:21.567972 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:21.567975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:21.567977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:21.568094 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:21.568552 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:21.568555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:21.568557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:21.568775 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:21.568872 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:21.569740 (+ 868us) server_negotiation.cc:299] Negotiation successful
0619 02:56:21.569892 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:56:21.571664 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:21.553319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46649 (local address 127.22.230.2:38121)
0619 02:56:21.553969 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:21.554004 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:21.555728 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:21.556423 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:21.556432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:21.556893 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:21.557427 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:21.557443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.558432 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.558438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:21.560094 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:21.560107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:21.560976 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:21.560987 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:21.561217 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:21.561899 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:21.561924 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:21.564018 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:21.567200 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:21.567212 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:21.567231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:21.567854 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:21.568195 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:21.568202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:21.568209 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:21.568449 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:21.568895 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:21.568906 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:21.569350 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 02:56:21.569625 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 02:56:21.570178 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":252,"threads_started":1}
I20260619 02:56:21.573098 24345 heartbeater.cc:347] Connected to a master server at 127.22.230.62:46649
I20260619 02:56:21.573429 24345 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:21.574316 24345 heartbeater.cc:511] Master 127.22.230.62:46649 requested a full tablet report, sending...
I20260619 02:56:21.575937 24022 ts_manager.cc:194] Registered new tserver with Master: ca547841fb7c435685e8859805a530e7 (127.22.230.2:39823)
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:56:21.576884 24022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:38121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:21.599589 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46649
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:21.912770 24350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:21.913028 24350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:21.913210 24350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:21.920120 24350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:21.920253 24350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:21.920444 24350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:21.929872 24350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:21.931185 24350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:21.932510 24350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:21.944741 24355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:21.944710 24356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:21.944770 24358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:21.945436 24350 server_base.cc:1034] running on GCE node
I20260619 02:56:21.946050 24350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:21.946987 24350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:21.948248 24350 hybrid_clock.cc:648] HybridClock initialized: now 1781837781948209 us; error 58 us; skew 500 ppm
I20260619 02:56:21.948511 24350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:21 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:21.951381 24350 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:21.952632 24350 webserver.cc:460] Webserver started at http://127.22.230.3:43431/ using document root <none> and password file <none>
I20260619 02:56:21.953222 24350 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:21.953321 24350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:21.953564 24350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:21.956442 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "b390e75b102ce3069856a36e6c484dff"
server_key_iv: "30f64600fdd383856fb30cda3def9bc6"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.957130 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "b390e75b102ce3069856a36e6c484dff"
server_key_iv: "30f64600fdd383856fb30cda3def9bc6"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.961982 24350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:56:21.965224 24365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.966449 24350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:56:21.966640 24350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd"
format_stamp: "Formatted at 2026-06-19 02:56:21 on dist-test-slave-2gms"
server_key: "b390e75b102ce3069856a36e6c484dff"
server_key_iv: "30f64600fdd383856fb30cda3def9bc6"
server_key_version: "encryptionkey@0"
I20260619 02:56:21.966809 24350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:21.980757 24350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:21.984787 24350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:21.985014 24350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:21.985826 24350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:21.987157 24350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:21.987226 24350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:21.987354 24350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:21.987425 24350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:22.018778 24350 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:33909
I20260619 02:56:22.018819 24477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:33909 every 8 connection(s)
I20260619 02:56:22.020020 24350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:22.025153 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24350
I20260619 02:56:22.025342 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:22.025923 23448 external_mini_cluster.cc:1428] Setting key 99bacd713a06c92cb27c8944466267d5
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837781, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:22.039702 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.022366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:57209 (local address 127.22.230.62:46649)
0619 02:56:22.022556 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.022580 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.023838 (+ 1258us) server_negotiation.cc:407] Connection header received
0619 02:56:22.026603 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.026608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.026700 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.026815 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.028404 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.029132 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.030776 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.031192 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.031321 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.034502 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.034507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.034510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.034540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.037099 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.037713 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.037716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.037718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.037820 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.038145 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.038148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.038149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.038380 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.038479 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.039150 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.039360 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:56:22.040676 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.022917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46649 (local address 127.22.230.3:57209)
0619 02:56:22.023683 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:22.023724 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:22.026276 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:22.027037 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:22.027045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:22.027708 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:22.028248 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.028263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.029291 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.029295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.030555 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.030621 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.031403 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.031411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.031652 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.032238 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:22.032258 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:22.034317 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:22.037212 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:22.037218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:22.037228 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:22.037622 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:22.037904 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:22.037907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:22.037909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:22.038063 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:22.038475 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:22.038479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:22.038828 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 02:56:22.039077 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 02:56:22.039554 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":373,"threads_started":1}
I20260619 02:56:22.041922 24478 heartbeater.cc:347] Connected to a master server at 127.22.230.62:46649
I20260619 02:56:22.042212 24478 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:22.042927 24478 heartbeater.cc:511] Master 127.22.230.62:46649 requested a full tablet report, sending...
I20260619 02:56:22.044174 24022 ts_manager.cc:194] Registered new tserver with Master: 76bdcf78b6ff4cc8bdb20b1763c381fd (127.22.230.3:33909)
I20260619 02:56:22.044934 24022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:57209
I20260619 02:56:22.052711 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:56:22.075598 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.059987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.22.230.62:46649)
0619 02:56:22.062747 (+ 2760us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.062752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.062771 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:56:22.062849 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.062853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.062927 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.063033 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.064944 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.066218 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.067631 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.068036 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.068170 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.070552 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.070558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.070563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.070636 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.073034 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.073658 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.073664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.073668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.073807 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.074167 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.074174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.074178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.074469 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.074765 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.075065 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.075282 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2616}
I20260619 02:56:22.082160 24022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:22.089412 24022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:22.110584 24211 heartbeater.cc:503] Master 127.22.230.62:46649 was elected leader, sending a full tablet report...
I20260619 02:56:22.121742 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.100476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.22.230.1:46727)
0619 02:56:22.112053 (+ 11577us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.112064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.113302 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 02:56:22.113714 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.113723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.114013 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.114172 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:22.115179 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.116894 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.119245 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.120295 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.120424 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.120629 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.121268 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.121445 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11420,"thread_start_us":11268,"threads_started":1}
I20260619 02:56:22.121755 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.100340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:46727 (local address 127.0.0.1:56476)
0619 02:56:22.113151 (+ 12811us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:22.113199 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:22.113470 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:22.114472 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:22.114481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:22.114517 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:22.114949 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.114966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.117054 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.117060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.119097 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.119108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.120798 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.120804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.121039 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.121075 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:56:22.121200 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:56:22.121376 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12568,"mutex_wait_us":2,"thread_start_us":13018,"threads_started":1}
I20260619 02:56:22.123414 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.100819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.22.230.3:33909)
0619 02:56:22.112489 (+ 11670us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.112497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.114794 (+ 2297us) server_negotiation.cc:407] Connection header received
0619 02:56:22.114921 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.114931 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.115155 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.116069 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:22.116350 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.117889 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.120990 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.122403 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.122539 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.122774 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.122971 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.123121 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11501,"thread_start_us":11669,"threads_started":1}
I20260619 02:56:22.123449 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.100639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:33909 (local address 127.0.0.1:43916)
0619 02:56:22.113627 (+ 12988us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:22.113645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:22.114066 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:22.115369 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:22.115373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:22.115401 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:22.115863 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.115876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.118020 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.118030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.120801 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.120821 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.122624 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.122631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.122792 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.122825 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:56:22.122935 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:56:22.123054 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12851,"mutex_wait_us":4,"thread_start_us":12132,"threads_started":1}
I20260619 02:56:22.123922 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.113894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:39823 (local address 127.0.0.1:54402)
0619 02:56:22.115214 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:22.115227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:22.115359 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:22.115887 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:22.115891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:22.115920 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:22.116226 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.116234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.117862 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.117870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.121040 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.121052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.123123 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.123129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.123347 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.123377 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:56:22.123476 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:56:22.123585 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"mutex_wait_us":6,"thread_start_us":1053,"threads_started":1}
I20260619 02:56:22.124199 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.113898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.22.230.2:39823)
0619 02:56:22.114796 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.114806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.115313 (+ 507us) server_negotiation.cc:407] Connection header received
0619 02:56:22.115518 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.115523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.115716 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.115908 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:22.116367 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.117724 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.121218 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.122697 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.122821 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.122981 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.123791 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.123945 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":651,"threads_started":1}
I20260619 02:56:22.126060 24146 tablet_service.cc:1467] Processing CreateTablet for tablet 59c55ba9504f47d6a1ce89d90353c899 (DEFAULT_TABLE table=test-table [id=e345639442c349f7a6415fdb663dd2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:22.126555 24413 tablet_service.cc:1467] Processing CreateTablet for tablet 59c55ba9504f47d6a1ce89d90353c899 (DEFAULT_TABLE table=test-table [id=e345639442c349f7a6415fdb663dd2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:22.127617 24146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c55ba9504f47d6a1ce89d90353c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:22.127815 24413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c55ba9504f47d6a1ce89d90353c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:22.127943 24280 tablet_service.cc:1467] Processing CreateTablet for tablet 59c55ba9504f47d6a1ce89d90353c899 (DEFAULT_TABLE table=test-table [id=e345639442c349f7a6415fdb663dd2f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:22.129366 24280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c55ba9504f47d6a1ce89d90353c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:22.140326 24496 tablet_bootstrap.cc:492] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: Bootstrap starting.
I20260619 02:56:22.143582 24496 tablet_bootstrap.cc:654] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:22.144578 24496 log.cc:824] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:22.147462 24498 tablet_bootstrap.cc:492] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: Bootstrap starting.
I20260619 02:56:22.148401 24497 tablet_bootstrap.cc:492] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: Bootstrap starting.
I20260619 02:56:22.152676 24497 tablet_bootstrap.cc:654] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:22.153796 24497 log.cc:824] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:22.154121 24498 tablet_bootstrap.cc:654] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:22.155012 24496 tablet_bootstrap.cc:492] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: No bootstrap required, opened a new log
I20260619 02:56:22.155318 24498 log.cc:824] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:22.155349 24496 ts_tablet_manager.cc:1397] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260619 02:56:22.156901 24497 tablet_bootstrap.cc:492] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: No bootstrap required, opened a new log
I20260619 02:56:22.157140 24497 ts_tablet_manager.cc:1397] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 02:56:22.157732 24498 tablet_bootstrap.cc:492] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: No bootstrap required, opened a new log
I20260619 02:56:22.157968 24498 ts_tablet_manager.cc:1397] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260619 02:56:22.166065 24498 raft_consensus.cc:348] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.166428 24498 raft_consensus.cc:374] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:22.166538 24498 raft_consensus.cc:729] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca547841fb7c435685e8859805a530e7, State: Initialized, Role: FOLLOWER
I20260619 02:56:22.166356 24496 raft_consensus.cc:348] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.166815 24496 raft_consensus.cc:374] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:22.166952 24496 raft_consensus.cc:729] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76bdcf78b6ff4cc8bdb20b1763c381fd, State: Initialized, Role: FOLLOWER
I20260619 02:56:22.167275 24498 consensus_queue.cc:260] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.167534 24496 consensus_queue.cc:260] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.168002 24497 raft_consensus.cc:348] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.168395 24345 heartbeater.cc:503] Master 127.22.230.62:46649 was elected leader, sending a full tablet report...
I20260619 02:56:22.168480 24497 raft_consensus.cc:374] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:22.168586 24497 raft_consensus.cc:729] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7642baa8b69140c5857ce685743844a6, State: Initialized, Role: FOLLOWER
I20260619 02:56:22.169379 24497 consensus_queue.cc:260] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.170805 24498 ts_tablet_manager.cc:1428] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260619 02:56:22.174422 24497 ts_tablet_manager.cc:1428] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260619 02:56:22.175328 24478 heartbeater.cc:503] Master 127.22.230.62:46649 was elected leader, sending a full tablet report...
I20260619 02:56:22.175967 24496 ts_tablet_manager.cc:1428] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: Time spent starting tablet: real 0.020s user 0.006s sys 0.014s
W20260619 02:56:22.271593 24479 tablet.cc:2369] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:56:22.302384 24346 tablet.cc:2369] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:56:22.333421 24212 tablet.cc:2369] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:56:22.434314 24504 raft_consensus.cc:482] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:22.434656 24504 raft_consensus.cc:504] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.436720 24504 leader_election.cc:288] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76bdcf78b6ff4cc8bdb20b1763c381fd (127.22.230.3:33909), ca547841fb7c435685e8859805a530e7 (127.22.230.2:39823)
I20260619 02:56:22.444334 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.437174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:43753 (local address 127.22.230.3:33909)
0619 02:56:22.437416 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.437425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.437823 (+ 398us) server_negotiation.cc:407] Connection header received
0619 02:56:22.438065 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.438073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.438177 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.438296 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:22.438861 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.440421 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.442095 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.443309 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.443482 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.443560 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.443957 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.444073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:56:22.444432 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.437097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:33909 (local address 127.22.230.1:43753)
0619 02:56:22.437694 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:22.437726 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:22.437899 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:22.438347 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:22.438354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:22.438380 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:22.438726 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.438734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.440559 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.440562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.441944 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.441955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.443528 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.443534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.443771 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.443798 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 02:56:22.443937 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 02:56:22.444067 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":36,"thread_start_us":351,"threads_started":1}
I20260619 02:56:22.445326 24433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c55ba9504f47d6a1ce89d90353c899" candidate_uuid: "7642baa8b69140c5857ce685743844a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" is_pre_election: true
I20260619 02:56:22.445798 24433 raft_consensus.cc:2454] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7642baa8b69140c5857ce685743844a6 in term 0.
I20260619 02:56:22.446518 24101 leader_election.cc:302] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7642baa8b69140c5857ce685743844a6, 76bdcf78b6ff4cc8bdb20b1763c381fd; no voters:
I20260619 02:56:22.446982 24504 raft_consensus.cc:2781] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:22.447568 24504 raft_consensus.cc:482] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:22.447674 24504 raft_consensus.cc:3037] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:22.449222 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.438106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:36613 (local address 127.22.230.2:39823)
0619 02:56:22.438323 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.438328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.443116 (+ 4788us) server_negotiation.cc:407] Connection header received
0619 02:56:22.443299 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.443304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.443367 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.443459 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:22.444078 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.444922 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.446544 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.448358 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.448517 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.448586 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.448853 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.448980 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:56:22.449231 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.437935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:39823 (local address 127.22.230.1:36613)
0619 02:56:22.443016 (+ 5081us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:22.443030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:22.443163 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:22.443476 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:22.443480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:22.443502 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:22.443936 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.443943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.445049 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.445055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.446400 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:22.446411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.448563 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.448567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:22.448697 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.448720 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:56:22.448804 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 02:56:22.448896 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4996,"mutex_wait_us":20,"thread_start_us":4843,"threads_started":1}
I20260619 02:56:22.450186 24300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c55ba9504f47d6a1ce89d90353c899" candidate_uuid: "7642baa8b69140c5857ce685743844a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca547841fb7c435685e8859805a530e7" is_pre_election: true
I20260619 02:56:22.450557 24504 raft_consensus.cc:504] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.450722 24300 raft_consensus.cc:2454] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7642baa8b69140c5857ce685743844a6 in term 0.
I20260619 02:56:22.451304 24504 leader_election.cc:288] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [CANDIDATE]: Term 1 election: Requested vote from peers 76bdcf78b6ff4cc8bdb20b1763c381fd (127.22.230.3:33909), ca547841fb7c435685e8859805a530e7 (127.22.230.2:39823)
I20260619 02:56:22.451572 24433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c55ba9504f47d6a1ce89d90353c899" candidate_uuid: "7642baa8b69140c5857ce685743844a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd"
I20260619 02:56:22.451787 24433 raft_consensus.cc:3037] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:22.452042 24300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c55ba9504f47d6a1ce89d90353c899" candidate_uuid: "7642baa8b69140c5857ce685743844a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca547841fb7c435685e8859805a530e7"
I20260619 02:56:22.452225 24300 raft_consensus.cc:3037] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:22.455104 24433 raft_consensus.cc:2454] T 59c55ba9504f47d6a1ce89d90353c899 P 76bdcf78b6ff4cc8bdb20b1763c381fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7642baa8b69140c5857ce685743844a6 in term 1.
I20260619 02:56:22.455282 24300 raft_consensus.cc:2454] T 59c55ba9504f47d6a1ce89d90353c899 P ca547841fb7c435685e8859805a530e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7642baa8b69140c5857ce685743844a6 in term 1.
I20260619 02:56:22.455511 24101 leader_election.cc:302] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7642baa8b69140c5857ce685743844a6, 76bdcf78b6ff4cc8bdb20b1763c381fd; no voters:
I20260619 02:56:22.455798 24504 raft_consensus.cc:2781] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:22.456248 24504 raft_consensus.cc:686] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [term 1 LEADER]: Becoming Leader. State: Replica: 7642baa8b69140c5857ce685743844a6, State: Running, Role: LEADER
I20260619 02:56:22.456794 24504 consensus_queue.cc:237] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } }
I20260619 02:56:22.461704 24022 catalog_manager.cc:5495] T 59c55ba9504f47d6a1ce89d90353c899 P 7642baa8b69140c5857ce685743844a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7642baa8b69140c5857ce685743844a6 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "7642baa8b69140c5857ce685743844a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7642baa8b69140c5857ce685743844a6" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76bdcf78b6ff4cc8bdb20b1763c381fd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 33909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca547841fb7c435685e8859805a530e7" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39823 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:22.491235 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.477205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.22.230.1:46727)
0619 02:56:22.477380 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.477385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.477867 (+ 482us) server_negotiation.cc:407] Connection header received
0619 02:56:22.477974 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.477977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.478034 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.478196 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.479771 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.480490 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.481658 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.482045 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.482131 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.485741 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.485752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.485766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.485794 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.489107 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.489659 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.489666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.489668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.489799 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.490082 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.490085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.490087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.490382 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.490632 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.490856 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.491013 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:22.538403 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.522967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.22.230.1:46727)
0619 02:56:22.523185 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.523190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.523444 (+ 254us) server_negotiation.cc:407] Connection header received
0619 02:56:22.523622 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.523626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.523688 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.523769 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.525376 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.526098 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.528031 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.528408 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.528503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.532690 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.532695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.532699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.532728 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.536076 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.536667 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.536672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.536676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.536814 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.537160 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.537165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.537168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.537478 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.537689 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.537870 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.538034 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:22.582422 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.568798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.22.230.1:46727)
0619 02:56:22.569026 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.569032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.569278 (+ 246us) server_negotiation.cc:407] Connection header received
0619 02:56:22.569408 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.569412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.569482 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.569604 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.571673 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.572416 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.573544 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.573931 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.574021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.577734 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.577739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.577742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.577772 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.580380 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.581008 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.581012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.581014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.581121 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.581409 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.581412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.581414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.581638 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.581830 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.582039 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.582182 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 02:56:22.584187 24126 tablet_service.cc:2908] Rejecting scan request for tablet 59c55ba9504f47d6a1ce89d90353c899: Uninitialized: clean time has not yet been initialized
I20260619 02:56:22.587975 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24083
I20260619 02:56:22.599896 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.588873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.22.230.1:46727)
0619 02:56:22.589013 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.589018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.589410 (+ 392us) server_negotiation.cc:407] Connection header received
0619 02:56:22.589528 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.589532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.589584 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.589676 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.590835 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.591567 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.592720 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.593111 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.593202 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.595328 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.595333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.595336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.595362 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.597839 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.598378 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.598381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.598384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.598504 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.598831 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.598834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.598836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.599072 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.599239 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.599484 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.599648 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:56:22.600513 24206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:22.691845 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24083
I20260619 02:56:22.709020 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24217
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:22.723171 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.709697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.22.230.2:39823)
0619 02:56:22.709869 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.709877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.709957 (+ 80us) server_negotiation.cc:407] Connection header received
0619 02:56:22.710164 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.710169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.710249 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.710355 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.711556 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.712286 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.713550 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.713945 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.714077 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.717955 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.717967 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.717982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.718014 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.720530 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.721166 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.721181 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.721183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.721305 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.721678 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.721684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.721687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.722023 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.722322 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.722643 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.722859 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:56:22.723773 24340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:22.811031 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24217
I20260619 02:56:22.828119 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24350
Jun 19 02:56:22 dist-test-slave-2gms krb5kdc[23977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:22.842231 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:22.828901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.22.230.3:33909)
0619 02:56:22.829090 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:22.829095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:22.829114 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:56:22.829343 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:22.829353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:22.829415 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:22.829520 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:22.830764 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.831511 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.832807 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:22.833195 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:22.833330 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:22.837373 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:22.837388 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:22.837404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:22.837434 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:22.840088 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.840648 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.840655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.840657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.840769 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:22.841070 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:22.841073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:22.841075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:22.841391 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:22.841588 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:22.841802 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:56:22.841969 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:56:22.842991 24473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:22.928534 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24350
I20260619 02:56:22.944469 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 23990
I20260619 02:56:22.945626 24052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:23.049403 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 23990
2026-06-19T02:56:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4175 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:23 dist-test-slave-2gms krb5kdc[24526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:23 dist-test-slave-2gms krb5kdc[24526](info): set up 2 sockets
Jun 19 02:56:23 dist-test-slave-2gms krb5kdc[24526](info): commencing operation
krb5kdc: starting...
W20260619 02:56:24.134528 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:24 dist-test-slave-2gms krb5kdc[24526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:24.310076 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:34229
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:34229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:24.617789 24539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:24.618072 24539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:24.618235 24539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:24.625598 24539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:24.625697 24539 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:56:24.625769 24539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:24.625866 24539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:24.625953 24539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:24.626075 24539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:24.636113 24539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:34229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:34229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:24.637373 24539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:24.638685 24539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:24.650486 24545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:24.650558 24547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:24.650558 24544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:24.651599 24539 server_base.cc:1034] running on GCE node
I20260619 02:56:24.652331 24539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:24.653702 24539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:24.654913 24539 hybrid_clock.cc:648] HybridClock initialized: now 1781837784654854 us; error 31 us; skew 500 ppm
I20260619 02:56:24.655272 24539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:24 dist-test-slave-2gms krb5kdc[24526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:24.659657 24539 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:24.661489 24539 webserver.cc:460] Webserver started at http://127.22.230.62:46759/ using document root <none> and password file <none>
I20260619 02:56:24.662362 24539 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:24.662489 24539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:24.662920 24539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:24.666244 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "16df71b4970e48eab5ab472972af276c"
format_stamp: "Formatted at 2026-06-19 02:56:24 on dist-test-slave-2gms"
server_key: "110fe4fc68503cf53197fe8a5013f449"
server_key_iv: "f602eabb73e2b934ac1ea9d6f7e75ec7"
server_key_version: "encryptionkey@0"
I20260619 02:56:24.667096 24539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "16df71b4970e48eab5ab472972af276c"
format_stamp: "Formatted at 2026-06-19 02:56:24 on dist-test-slave-2gms"
server_key: "110fe4fc68503cf53197fe8a5013f449"
server_key_iv: "f602eabb73e2b934ac1ea9d6f7e75ec7"
server_key_version: "encryptionkey@0"
I20260619 02:56:24.671976 24539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 02:56:24.675258 24554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:24.676558 24539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:24.676734 24539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "16df71b4970e48eab5ab472972af276c"
format_stamp: "Formatted at 2026-06-19 02:56:24 on dist-test-slave-2gms"
server_key: "110fe4fc68503cf53197fe8a5013f449"
server_key_iv: "f602eabb73e2b934ac1ea9d6f7e75ec7"
server_key_version: "encryptionkey@0"
I20260619 02:56:24.676934 24539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:24.691468 24539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:24.695221 24539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:24.695477 24539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:24.715878 24539 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:34229
I20260619 02:56:24.715901 24615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:34229 every 8 connection(s)
I20260619 02:56:24.717219 24539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:24.720822 24616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:24.725674 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24539
I20260619 02:56:24.725939 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:24.726537 23448 external_mini_cluster.cc:1428] Setting key 3b25ced6427a16df1bbdd4a07a39de63
I20260619 02:56:24.731959 24616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: Bootstrap starting.
I20260619 02:56:24.735492 24616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:24.736552 24616 log.cc:824] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:24.739370 24616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: No bootstrap required, opened a new log
Jun 19 02:56:24 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:24.749197 24616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16df71b4970e48eab5ab472972af276c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34229 } }
I20260619 02:56:24.749617 24616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:24.749724 24616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16df71b4970e48eab5ab472972af276c, State: Initialized, Role: FOLLOWER
I20260619 02:56:24.750406 24616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16df71b4970e48eab5ab472972af276c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34229 } }
I20260619 02:56:24.750705 24616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:24.750845 24616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:24.751020 24616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:24.751067 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:24.728503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.22.230.62:34229)
0619 02:56:24.729372 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:24.729389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:24.729440 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:56:24.731740 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:24.731766 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:24.732254 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:24.732666 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:24.733858 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:24.735818 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:24.737124 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:24.737943 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:24.738122 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:24.741920 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:24.741930 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:24.741947 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:24.741995 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:24.745892 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:24.746496 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:24.746506 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:24.746518 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:24.746764 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:24.747091 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:24.747097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:24.747101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:24.747814 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:24.748092 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:24.749287 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0619 02:56:24.749705 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":377,"threads_started":1}
I20260619 02:56:24.753865 24616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16df71b4970e48eab5ab472972af276c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34229 } }
I20260619 02:56:24.754279 24616 leader_election.cc:302] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16df71b4970e48eab5ab472972af276c; no voters:
I20260619 02:56:24.754854 24616 leader_election.cc:288] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:24.754992 24621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:24.755659 24621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [term 1 LEADER]: Becoming Leader. State: Replica: 16df71b4970e48eab5ab472972af276c, State: Running, Role: LEADER
I20260619 02:56:24.756214 24621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16df71b4970e48eab5ab472972af276c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34229 } }
I20260619 02:56:24.756913 24616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:24.758827 24623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16df71b4970e48eab5ab472972af276c. Latest consensus state: current_term: 1 leader_uuid: "16df71b4970e48eab5ab472972af276c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16df71b4970e48eab5ab472972af276c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34229 } } }
I20260619 02:56:24.759255 24623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:24.758915 24622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16df71b4970e48eab5ab472972af276c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16df71b4970e48eab5ab472972af276c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34229 } } }
I20260619 02:56:24.759622 24622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:24.760425 24629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:24.767407 24629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:24.775837 24629 catalog_manager.cc:1347] Generated new cluster ID: b3a9f3b6d8834125959e51d5574d3e01
I20260619 02:56:24.775969 24629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:24.786656 24629 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:24.787853 24629 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:24.794175 24629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: Generated new TSK 0
I20260619 02:56:24.794955 24629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:24.851791 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34229
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:25.166373 24642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:25.166688 24642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:25.166855 24642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:25.173923 24642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:25.174041 24642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:25.174192 24642 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:56:25.174321 24642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:25.184345 24642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:34229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:25.185627 24642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:25.186961 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:25.199237 24647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:25.199263 24648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:25.199488 24650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:25.200459 24642 server_base.cc:1034] running on GCE node
I20260619 02:56:25.200958 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:25.201726 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:25.202965 24642 hybrid_clock.cc:648] HybridClock initialized: now 1781837785202910 us; error 54 us; skew 500 ppm
I20260619 02:56:25.203241 24642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:25 dist-test-slave-2gms krb5kdc[24526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:25.206166 24642 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:25.207554 24642 webserver.cc:460] Webserver started at http://127.22.230.1:40801/ using document root <none> and password file <none>
I20260619 02:56:25.208206 24642 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:25.208348 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:25.208652 24642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:25.211457 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "7d0a59fa929c4d259e2370d754790f74"
format_stamp: "Formatted at 2026-06-19 02:56:25 on dist-test-slave-2gms"
server_key: "c5b95c963aab62ae509cd04960efff44"
server_key_iv: "034ed3343b855fa3bdfb7e69d3374616"
server_key_version: "encryptionkey@0"
I20260619 02:56:25.212220 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "7d0a59fa929c4d259e2370d754790f74"
format_stamp: "Formatted at 2026-06-19 02:56:25 on dist-test-slave-2gms"
server_key: "c5b95c963aab62ae509cd04960efff44"
server_key_iv: "034ed3343b855fa3bdfb7e69d3374616"
server_key_version: "encryptionkey@0"
I20260619 02:56:25.217085 24642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:56:25.220357 24657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:25.221669 24642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:56:25.221840 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "7d0a59fa929c4d259e2370d754790f74"
format_stamp: "Formatted at 2026-06-19 02:56:25 on dist-test-slave-2gms"
server_key: "c5b95c963aab62ae509cd04960efff44"
server_key_iv: "034ed3343b855fa3bdfb7e69d3374616"
server_key_version: "encryptionkey@0"
I20260619 02:56:25.222052 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:25.236505 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:25.240296 24642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:25.240556 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:25.243093 24642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:25.243189 24642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:25.243393 24642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:25.243479 24642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:56:25 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:25.266073 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.243528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:37801 (local address 127.22.230.62:34229)
0619 02:56:25.243697 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:25.243704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:25.245088 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 02:56:25.247195 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:25.247201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:25.247284 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:25.247390 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:25.249354 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.250303 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.252897 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.253650 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.253822 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.258509 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:25.258519 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:25.258526 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:25.258606 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:25.262723 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.263405 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.263414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.263420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.263632 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.264044 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.264052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.264058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.264489 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:25.264660 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:25.265273 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 02:56:25.265580 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:56:25.266556 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.244159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.22.230.1:37801)
0619 02:56:25.244943 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:25.244980 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:25.246901 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:25.247605 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:25.247613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:25.248386 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:25.249185 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.249205 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.250499 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.250505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.252734 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.252752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.253871 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.253876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.254043 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.254723 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:25.254755 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:25.258273 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:25.262868 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.262880 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.262890 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.263273 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.263747 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.263750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.263752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.263925 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.264623 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:25.264628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:25.264923 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 02:56:25.265120 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 02:56:25.265451 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":329,"threads_started":1}
I20260619 02:56:25.288254 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.273126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:34355 (local address 127.22.230.62:34229)
0619 02:56:25.273295 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:25.273300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:25.273855 (+ 555us) server_negotiation.cc:407] Connection header received
0619 02:56:25.273956 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:25.273964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:25.274086 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:25.274226 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:25.275265 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.276514 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.279161 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.279560 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.279653 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.282094 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:25.282098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:25.282102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:25.282127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:25.285380 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.286264 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.286269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.286273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.286445 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.286879 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.286882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.286884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.287156 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:25.287269 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:25.287804 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 02:56:25.288024 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:56:25.288722 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.273262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.22.230.1:34355)
0619 02:56:25.273753 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:25.273767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:25.273861 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:25.274246 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:25.274249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:25.274707 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:25.275094 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.275111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.276710 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.276716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.279002 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.279020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.279752 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.279761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.280015 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.280638 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:25.280668 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:25.281928 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:25.285497 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.285505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.285509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.286124 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.286588 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.286593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.286600 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.286746 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.287271 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:25.287281 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:25.287488 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 02:56:25.287659 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 02:56:25.288080 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
I20260619 02:56:25.299697 24642 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40835
I20260619 02:56:25.299736 24776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40835 every 8 connection(s)
I20260619 02:56:25.300846 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:25.302759 24777 heartbeater.cc:347] Connected to a master server at 127.22.230.62:34229
I20260619 02:56:25.302971 24777 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:25.303586 24777 heartbeater.cc:511] Master 127.22.230.62:34229 requested a full tablet report, sending...
I20260619 02:56:25.305377 24571 ts_manager.cc:194] Registered new tserver with Master: 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835)
I20260619 02:56:25.306849 24571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:37801
I20260619 02:56:25.309573 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24642
I20260619 02:56:25.309801 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:25.310393 23448 external_mini_cluster.cc:1428] Setting key ef9376bc108148847ab6fa634ac5d56e
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:25.349215 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34229
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:25.695777 24781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:25.696022 24781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:25.696169 24781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:25.703115 24781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:25.703207 24781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:25.703348 24781 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:56:25.703469 24781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:25.712774 24781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:34229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:25.713960 24781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:25.715229 24781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:25.729465 24787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:25.729440 24786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:25.730444 24789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:25.735908 24781 server_base.cc:1034] running on GCE node
I20260619 02:56:25.736508 24781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:25.737458 24781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:25.738713 24781 hybrid_clock.cc:648] HybridClock initialized: now 1781837785738639 us; error 49 us; skew 500 ppm
I20260619 02:56:25.738998 24781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:25 dist-test-slave-2gms krb5kdc[24526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:25.741946 24781 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:25.743569 24781 webserver.cc:460] Webserver started at http://127.22.230.2:33449/ using document root <none> and password file <none>
I20260619 02:56:25.744310 24781 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:25.744390 24781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:25.744735 24781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:25.747462 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "ec4d090a17754943a92f90e1dd25f791"
format_stamp: "Formatted at 2026-06-19 02:56:25 on dist-test-slave-2gms"
server_key: "d36677dbd99bd0f4307ae1e0f4e5fe1a"
server_key_iv: "ff4aa44107b1ac4beefe3a0fde37c78c"
server_key_version: "encryptionkey@0"
I20260619 02:56:25.748212 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "ec4d090a17754943a92f90e1dd25f791"
format_stamp: "Formatted at 2026-06-19 02:56:25 on dist-test-slave-2gms"
server_key: "d36677dbd99bd0f4307ae1e0f4e5fe1a"
server_key_iv: "ff4aa44107b1ac4beefe3a0fde37c78c"
server_key_version: "encryptionkey@0"
I20260619 02:56:25.752965 24781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:25.756325 24796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:25.757675 24781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:56:25.757833 24781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "ec4d090a17754943a92f90e1dd25f791"
format_stamp: "Formatted at 2026-06-19 02:56:25 on dist-test-slave-2gms"
server_key: "d36677dbd99bd0f4307ae1e0f4e5fe1a"
server_key_iv: "ff4aa44107b1ac4beefe3a0fde37c78c"
server_key_version: "encryptionkey@0"
I20260619 02:56:25.758020 24781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:25.771538 24781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:25.775391 24781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:25.775620 24781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:25.778107 24781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:25.778205 24781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:25.778364 24781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:25.778437 24781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:56:25 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837785, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:25.800336 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.778625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:39195 (local address 127.22.230.62:34229)
0619 02:56:25.778842 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:25.778850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:25.780729 (+ 1879us) server_negotiation.cc:407] Connection header received
0619 02:56:25.782878 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:25.782886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:25.782973 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:25.783047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:25.785407 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.786743 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.789342 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.790139 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.790296 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.794710 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:25.794715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:25.794718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:25.794748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:25.797187 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.797857 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.797864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.797867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.797986 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.798383 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.798390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.798395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.798717 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:25.798912 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:25.799934 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0619 02:56:25.800103 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 02:56:25.802433 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.779212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.22.230.2:39195)
0619 02:56:25.780570 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:25.780617 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:25.782483 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:25.783588 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:25.783601 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:25.784328 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:25.785176 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.785207 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.786936 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.786948 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.789149 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.789190 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.790355 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.790360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.790567 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.791347 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:25.791385 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:25.794429 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:25.797329 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.797337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.797350 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.797742 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.798097 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.798102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.798106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.798269 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.798895 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:25.798905 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:25.799465 (+ 560us) client_negotiation.cc:769] Sending connection context
0619 02:56:25.799879 (+ 414us) client_negotiation.cc:240] Negotiation successful
0619 02:56:25.800550 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":340,"threads_started":1}
I20260619 02:56:25.823733 24781 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:39967
I20260619 02:56:25.823736 24915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:39967 every 8 connection(s)
I20260619 02:56:25.824942 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.809281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:53721 (local address 127.22.230.62:34229)
0619 02:56:25.809811 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:25.809819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:25.810290 (+ 471us) server_negotiation.cc:407] Connection header received
0619 02:56:25.810613 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:25.810619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:25.810689 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:25.810743 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:25.812339 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.813392 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.815492 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.815900 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.816039 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.818294 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:25.818302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:25.818308 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:25.818352 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:25.822028 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.822895 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.822902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.822911 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.823021 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:25.823406 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:25.823414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:25.823417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:25.823707 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:25.823891 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:25.824348 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 02:56:25.824572 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382}
I20260619 02:56:25.825067 24874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:25.809454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.22.230.2:53721)
0619 02:56:25.810179 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:25.810203 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:25.810412 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:25.810994 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:25.811001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:25.811691 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:25.812139 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.812155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.813584 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.813587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.815332 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:25.815344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:25.816087 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:25.816092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:25.816269 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:25.816943 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:25.816972 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:25.818065 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:25.822164 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.822167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.822170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.822731 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.823138 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:25.823144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:25.823147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:25.823280 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:25.823852 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:25.823857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:25.824068 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 02:56:25.824260 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 02:56:25.824586 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":406,"threads_started":1}
I20260619 02:56:25.825194 24781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:25.827363 24916 heartbeater.cc:347] Connected to a master server at 127.22.230.62:34229
I20260619 02:56:25.828537 24916 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:25.829279 24916 heartbeater.cc:511] Master 127.22.230.62:34229 requested a full tablet report, sending...
I20260619 02:56:25.830547 24571 ts_manager.cc:194] Registered new tserver with Master: ec4d090a17754943a92f90e1dd25f791 (127.22.230.2:39967)
I20260619 02:56:25.831368 24571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:39195
I20260619 02:56:25.835461 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24781
I20260619 02:56:25.835672 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:25.836086 23448 external_mini_cluster.cc:1428] Setting key f94c5df1f3b1fade1a50cbcadecfd430
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:25.873821 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34229
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:26.192996 24920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:26.193259 24920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:26.193414 24920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:26.200728 24920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:26.200832 24920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:26.201004 24920 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:56:26.201116 24920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:26.210800 24920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:34229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:26.211982 24920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:26.213231 24920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:26.226068 24926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:26.226125 24928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:26.226079 24925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:26.226662 24920 server_base.cc:1034] running on GCE node
I20260619 02:56:26.227308 24920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:26.228143 24920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:26.229387 24920 hybrid_clock.cc:648] HybridClock initialized: now 1781837786229329 us; error 39 us; skew 500 ppm
I20260619 02:56:26.229681 24920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:26 dist-test-slave-2gms krb5kdc[24526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:26.232775 24920 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:26.234050 24920 webserver.cc:460] Webserver started at http://127.22.230.3:35179/ using document root <none> and password file <none>
I20260619 02:56:26.234807 24920 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:26.234928 24920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:26.235244 24920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:26.237944 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "2cbfef5bf5a44825824fa589da3176ce"
format_stamp: "Formatted at 2026-06-19 02:56:26 on dist-test-slave-2gms"
server_key: "eecdcac3bb25e741838aaa1fbd81aef1"
server_key_iv: "202e8e8663c1cec5befb1b01a7356601"
server_key_version: "encryptionkey@0"
I20260619 02:56:26.238816 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "2cbfef5bf5a44825824fa589da3176ce"
format_stamp: "Formatted at 2026-06-19 02:56:26 on dist-test-slave-2gms"
server_key: "eecdcac3bb25e741838aaa1fbd81aef1"
server_key_iv: "202e8e8663c1cec5befb1b01a7356601"
server_key_version: "encryptionkey@0"
I20260619 02:56:26.243618 24920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:56:26.246832 24935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:26.248193 24920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:56:26.248355 24920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "2cbfef5bf5a44825824fa589da3176ce"
format_stamp: "Formatted at 2026-06-19 02:56:26 on dist-test-slave-2gms"
server_key: "eecdcac3bb25e741838aaa1fbd81aef1"
server_key_iv: "202e8e8663c1cec5befb1b01a7356601"
server_key_version: "encryptionkey@0"
I20260619 02:56:26.248553 24920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:26.261847 24920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:26.265479 24920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:26.265733 24920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:26.268383 24920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:26.268481 24920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:26.268640 24920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:26.268748 24920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 02:56:26 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:26.290457 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.268754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:34887 (local address 127.22.230.62:34229)
0619 02:56:26.268927 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.268932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.270318 (+ 1386us) server_negotiation.cc:407] Connection header received
0619 02:56:26.272419 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.272428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.272521 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.272664 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:26.274715 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.275984 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.277556 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.278441 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.278620 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.283411 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:26.283420 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:26.283428 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:26.283477 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:26.287276 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.288069 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.288076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.288081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.288216 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.288644 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.288649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.288652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.288944 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:26.289096 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.289826 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.290046 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:56:26.291481 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.269409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.22.230.3:34887)
0619 02:56:26.270158 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.270198 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.271998 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.272931 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.272947 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.273687 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:26.274478 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.274496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.276142 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.276147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.277419 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.277429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.278724 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.278735 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.279042 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.279925 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:26.279959 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:26.283149 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:26.287418 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:26.287426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:26.287446 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:26.287892 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:26.288334 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:26.288340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:26.288343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:26.288525 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:26.289069 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:26.289077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:26.289411 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.289729 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.290220 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":346,"threads_started":1}
I20260619 02:56:26.309333 24777 heartbeater.cc:503] Master 127.22.230.62:34229 was elected leader, sending a full tablet report...
I20260619 02:56:26.314410 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.298071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:59719 (local address 127.22.230.62:34229)
0619 02:56:26.298294 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.298303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.298967 (+ 664us) server_negotiation.cc:407] Connection header received
0619 02:56:26.299117 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.299121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.299221 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.299327 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:26.300279 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.300963 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.302712 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.303538 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.303690 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.306172 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:26.306181 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:26.306187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:26.306227 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:26.310705 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.311769 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.311778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.311798 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.311980 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.312492 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.312497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.312500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.312881 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:26.313099 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.313681 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.313952 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:56:26.314604 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.298059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.22.230.3:59719)
0619 02:56:26.298841 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.298862 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.299006 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.299365 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.299372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.299820 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:26.300155 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.300167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.301090 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.301095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.302531 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.302545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.303777 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.303784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.303997 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.304848 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:26.304869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:26.305971 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:26.310900 (+ 4929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:26.310909 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:26.310916 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:26.311580 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:26.312141 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:26.312155 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:26.312161 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:26.312351 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:26.313073 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:26.313081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:26.313341 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.313530 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.313965 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260619 02:56:26.325093 24920 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40803
I20260619 02:56:26.325116 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40803 every 8 connection(s)
I20260619 02:56:26.326507 24920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:26.328569 25055 heartbeater.cc:347] Connected to a master server at 127.22.230.62:34229
I20260619 02:56:26.328756 25055 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:26.329303 25055 heartbeater.cc:511] Master 127.22.230.62:34229 requested a full tablet report, sending...
I20260619 02:56:26.330442 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 24920
I20260619 02:56:26.330667 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:26.330650 24571 ts_manager.cc:194] Registered new tserver with Master: 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803)
I20260619 02:56:26.331179 23448 external_mini_cluster.cc:1428] Setting key c4e7e0e9910fcd6ba9a0803597ab84db
I20260619 02:56:26.331744 24571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:34887
I20260619 02:56:26.333968 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:56:26 dist-test-slave-2gms krb5kdc[24526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:56:26 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:26.377615 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.364698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.22.230.62:34229)
0619 02:56:26.364865 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.364870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.365202 (+ 332us) server_negotiation.cc:407] Connection header received
0619 02:56:26.365356 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.365360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.365423 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.365514 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:26.366860 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.367593 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.368903 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.369298 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.369392 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.373074 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:26.373079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:26.373082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:26.373112 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:26.375644 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.376190 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.376194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.376196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.376308 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.376597 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.376600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.376602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.376829 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:26.376989 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.377243 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.377389 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:56:26.381533 24571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:26.388269 24571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:26.419849 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.399734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40803 (local address 127.0.0.1:56930)
0619 02:56:26.411896 (+ 12162us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.411950 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.412155 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.413032 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.413040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.413068 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:26.413498 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.413511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.415446 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.415451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.417405 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.417416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.418729 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.418736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.419067 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.419114 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.419244 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.419447 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11856,"mutex_wait_us":62,"thread_start_us":12602,"threads_started":1}
I20260619 02:56:26.420683 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.400057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.22.230.3:40803)
0619 02:56:26.411081 (+ 11024us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.411094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.412388 (+ 1294us) server_negotiation.cc:407] Connection header received
0619 02:56:26.412503 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.412513 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.412797 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.412989 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:26.413725 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.415319 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.417586 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.418501 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.418672 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.418925 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.420220 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.420375 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10811,"thread_start_us":11037,"threads_started":1}
I20260619 02:56:26.421491 25071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.400308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:39967 (local address 127.0.0.1:58940)
0619 02:56:26.413672 (+ 13364us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.413695 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.413806 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.414374 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.414380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.414406 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:26.414892 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.414904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.416571 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.416580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.419127 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.419135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.420697 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.420704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.420863 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.420894 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.420992 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.421126 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13221,"mutex_wait_us":177,"thread_start_us":13177,"threads_started":1}
I20260619 02:56:26.421605 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.401193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.22.230.2:39967)
0619 02:56:26.413193 (+ 12000us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.413202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.413771 (+ 569us) server_negotiation.cc:407] Connection header received
0619 02:56:26.413935 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.413940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.414122 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.414302 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:26.415065 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.416417 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.419277 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.420482 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.420632 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.420824 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.421142 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.421313 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11844,"thread_start_us":11703,"threads_started":1}
I20260619 02:56:26.422734 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.400832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.22.230.1:40835)
0619 02:56:26.413585 (+ 12753us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.413595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.413621 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:56:26.413707 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.413713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.413947 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.414156 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:26.414822 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.416308 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.419261 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.420659 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.420799 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.420985 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.422246 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.422404 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12603,"thread_start_us":12475,"threads_started":1}
I20260619 02:56:26.422927 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.399991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40835 (local address 127.0.0.1:33844)
0619 02:56:26.411998 (+ 12007us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.412022 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.412148 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.414173 (+ 2025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.414177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.414201 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:26.414663 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.414681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.416466 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.416473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.419111 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.419124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.421146 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.421152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.421344 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.421383 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.421465 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.421579 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11881,"mutex_wait_us":136,"thread_start_us":12153,"threads_started":1}
I20260619 02:56:26.425000 24846 tablet_service.cc:1467] Processing CreateTablet for tablet 34953dd7b3db43018efd86b5918db615 (DEFAULT_TABLE table=test-table [id=fe8240f421264b829d3ba019587fd9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:26.424983 24985 tablet_service.cc:1467] Processing CreateTablet for tablet 34953dd7b3db43018efd86b5918db615 (DEFAULT_TABLE table=test-table [id=fe8240f421264b829d3ba019587fd9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:26.425638 24707 tablet_service.cc:1467] Processing CreateTablet for tablet 34953dd7b3db43018efd86b5918db615 (DEFAULT_TABLE table=test-table [id=fe8240f421264b829d3ba019587fd9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:26.426409 24985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34953dd7b3db43018efd86b5918db615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.426411 24846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34953dd7b3db43018efd86b5918db615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.426874 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34953dd7b3db43018efd86b5918db615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.436996 25074 tablet_bootstrap.cc:492] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Bootstrap starting.
I20260619 02:56:26.440474 25075 tablet_bootstrap.cc:492] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Bootstrap starting.
I20260619 02:56:26.441184 25074 tablet_bootstrap.cc:654] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.442526 25074 log.cc:824] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:26.443915 25076 tablet_bootstrap.cc:492] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Bootstrap starting.
I20260619 02:56:26.444012 25075 tablet_bootstrap.cc:654] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.445190 25075 log.cc:824] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:26.446605 25074 tablet_bootstrap.cc:492] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: No bootstrap required, opened a new log
I20260619 02:56:26.446923 25074 ts_tablet_manager.cc:1397] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 02:56:26.448438 25075 tablet_bootstrap.cc:492] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: No bootstrap required, opened a new log
I20260619 02:56:26.448760 25075 ts_tablet_manager.cc:1397] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 02:56:26.448787 25076 tablet_bootstrap.cc:654] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.450368 25076 log.cc:824] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:26.456205 25075 raft_consensus.cc:348] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.456542 25075 raft_consensus.cc:374] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.456707 25075 raft_consensus.cc:729] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4d090a17754943a92f90e1dd25f791, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.457446 25075 consensus_queue.cc:260] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.459141 25076 tablet_bootstrap.cc:492] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: No bootstrap required, opened a new log
I20260619 02:56:26.459457 24916 heartbeater.cc:503] Master 127.22.230.62:34229 was elected leader, sending a full tablet report...
I20260619 02:56:26.459522 25074 raft_consensus.cc:348] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.459924 25074 raft_consensus.cc:374] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.459928 25075 ts_tablet_manager.cc:1428] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260619 02:56:26.460017 25074 raft_consensus.cc:729] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0a59fa929c4d259e2370d754790f74, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.460686 25074 consensus_queue.cc:260] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.461459 25076 ts_tablet_manager.cc:1397] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260619 02:56:26.467890 25074 ts_tablet_manager.cc:1428] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260619 02:56:26.472182 25076 raft_consensus.cc:348] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.472513 25076 raft_consensus.cc:374] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.472626 25076 raft_consensus.cc:729] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbfef5bf5a44825824fa589da3176ce, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.473096 25076 consensus_queue.cc:260] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.474018 25055 heartbeater.cc:503] Master 127.22.230.62:34229 was elected leader, sending a full tablet report...
I20260619 02:56:26.474393 25076 ts_tablet_manager.cc:1428] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260619 02:56:26.539034 25080 raft_consensus.cc:482] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:26.539397 25080 raft_consensus.cc:504] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.541260 25080 leader_election.cc:288] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803), 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835)
I20260619 02:56:26.550400 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.541728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:48919 (local address 127.22.230.3:40803)
0619 02:56:26.541921 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.541926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.543156 (+ 1230us) server_negotiation.cc:407] Connection header received
0619 02:56:26.543298 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.543306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.543378 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.543465 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:26.544069 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.545075 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.548043 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.549247 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.549417 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.549486 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.550018 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.550141 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:56:26.550400 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.541669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:55219 (local address 127.22.230.1:40835)
0619 02:56:26.541862 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.541869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.542868 (+ 999us) server_negotiation.cc:407] Connection header received
0619 02:56:26.543100 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.543110 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.543218 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.543330 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:26.543906 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.544986 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.548049 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.549249 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.549414 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.549487 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.550018 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.550141 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:56:26.550714 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.542047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40803 (local address 127.22.230.2:48919)
0619 02:56:26.543056 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.543070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.543181 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.543500 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.543506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.543532 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:26.543936 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.543951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.545192 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.545198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.547892 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.547901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.549602 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.549608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.549839 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.549887 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.549980 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.550210 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":353,"thread_start_us":685,"threads_started":1}
I20260619 02:56:26.550738 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.541438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40835 (local address 127.22.230.2:55219)
0619 02:56:26.542743 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.542759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.542953 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.543334 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.543337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.543352 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:26.543750 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.543766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.545103 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.545107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.547885 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.547901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.549589 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.549596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.549827 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.549883 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.549980 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.550210 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":202,"thread_start_us":817,"threads_started":1}
I20260619 02:56:26.551682 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34953dd7b3db43018efd86b5918db615" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbfef5bf5a44825824fa589da3176ce" is_pre_election: true
I20260619 02:56:26.551672 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34953dd7b3db43018efd86b5918db615" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0a59fa929c4d259e2370d754790f74" is_pre_election: true
I20260619 02:56:26.552243 24732 raft_consensus.cc:2454] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 0.
I20260619 02:56:26.552237 25010 raft_consensus.cc:2454] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 0.
W20260619 02:56:26.553140 24778 tablet.cc:2369] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:56:26.553308 24799 leader_election.cc:302] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbfef5bf5a44825824fa589da3176ce, ec4d090a17754943a92f90e1dd25f791; no voters:
I20260619 02:56:26.553759 25080 raft_consensus.cc:2781] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:26.553977 25080 raft_consensus.cc:482] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:26.554075 25080 raft_consensus.cc:3037] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:26.556154 25080 raft_consensus.cc:504] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.556646 25080 leader_election.cc:288] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 election: Requested vote from peers 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803), 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835)
I20260619 02:56:26.557073 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34953dd7b3db43018efd86b5918db615" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbfef5bf5a44825824fa589da3176ce"
I20260619 02:56:26.557113 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34953dd7b3db43018efd86b5918db615" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0a59fa929c4d259e2370d754790f74"
I20260619 02:56:26.557303 25010 raft_consensus.cc:3037] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:26.557377 24732 raft_consensus.cc:3037] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:26.559684 24732 raft_consensus.cc:2454] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 1.
I20260619 02:56:26.559684 25010 raft_consensus.cc:2454] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 1.
I20260619 02:56:26.560086 24799 leader_election.cc:302] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2cbfef5bf5a44825824fa589da3176ce, 7d0a59fa929c4d259e2370d754790f74, ec4d090a17754943a92f90e1dd25f791; no voters:
I20260619 02:56:26.560472 25080 raft_consensus.cc:2781] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:26.560927 25080 raft_consensus.cc:686] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 1 LEADER]: Becoming Leader. State: Replica: ec4d090a17754943a92f90e1dd25f791, State: Running, Role: LEADER
I20260619 02:56:26.561501 25080 consensus_queue.cc:237] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } }
I20260619 02:56:26.565822 24570 catalog_manager.cc:5495] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4d090a17754943a92f90e1dd25f791 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "ec4d090a17754943a92f90e1dd25f791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } health_report { overall_health: HEALTHY } } }
W20260619 02:56:26.577572 24917 tablet.cc:2369] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:56:26.577992 25056 tablet.cc:2369] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:56:26.606866 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.601676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.22.230.2:39967)
0619 02:56:26.601859 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.601863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.602040 (+ 177us) server_negotiation.cc:407] Connection header received
0619 02:56:26.602301 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.602305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.602385 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.602501 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:26.602871 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.603561 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.604772 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.605321 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.605413 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.605869 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:26.606358 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:26.606460 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.606550 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.606665 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:56:26.614621 25010 raft_consensus.cc:1261] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Refusing update from remote peer ec4d090a17754943a92f90e1dd25f791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:26.614617 24732 raft_consensus.cc:1261] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Refusing update from remote peer ec4d090a17754943a92f90e1dd25f791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:26.615631 25085 consensus_queue.cc:1035] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:26.615983 25080 consensus_queue.cc:1035] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:26.627378 25088 mvcc.cc:205] Tried to move back new op lower bound from 7298407573963358208 to 7298407573759791104. Current Snapshot: MvccSnapshot[applied={T|T < 7298407573963358208}]
I20260619 02:56:26.630147 25090 mvcc.cc:205] Tried to move back new op lower bound from 7298407573963358208 to 7298407573759791104. Current Snapshot: MvccSnapshot[applied={T|T < 7298407573963358208}]
I20260619 02:56:26.645466 24571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:26.645805 24571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:26.649600 24571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: Sending DeleteTablet for 3 replicas of tablet 34953dd7b3db43018efd86b5918db615
I20260619 02:56:26.650656 24707 tablet_service.cc:1514] Processing DeleteTablet for tablet 34953dd7b3db43018efd86b5918db615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:26 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:33844
I20260619 02:56:26.650731 24985 tablet_service.cc:1514] Processing DeleteTablet for tablet 34953dd7b3db43018efd86b5918db615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:26 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56930
I20260619 02:56:26.650954 24846 tablet_service.cc:1514] Processing DeleteTablet for tablet 34953dd7b3db43018efd86b5918db615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:26 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:58940
I20260619 02:56:26.655062 24571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:26.655925 24571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:26.666237 25099 tablet_replica.cc:330] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: stopping tablet replica
I20260619 02:56:26.666756 25099 raft_consensus.cc:2229] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:26.668057 24984 tablet_service.cc:1467] Processing CreateTablet for tablet a2bfb3d2a3e743f7aae5087994bd0743 (DEFAULT_TABLE table=test-table [id=ede8bd76c9354ecf8914f8fe56c202e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:26.668599 24984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bfb3d2a3e743f7aae5087994bd0743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.668957 24845 tablet_service.cc:1467] Processing CreateTablet for tablet a2bfb3d2a3e743f7aae5087994bd0743 (DEFAULT_TABLE table=test-table [id=ede8bd76c9354ecf8914f8fe56c202e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:26.669800 24845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bfb3d2a3e743f7aae5087994bd0743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.671440 25098 tablet_replica.cc:330] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: stopping tablet replica
I20260619 02:56:26.672142 25100 tablet_replica.cc:330] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: stopping tablet replica
I20260619 02:56:26.672623 25098 raft_consensus.cc:2229] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:26.673141 25100 raft_consensus.cc:2229] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:26.673834 25100 raft_consensus.cc:2258] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:26.674362 25098 raft_consensus.cc:2258] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:26.675068 24706 tablet_service.cc:1467] Processing CreateTablet for tablet a2bfb3d2a3e743f7aae5087994bd0743 (DEFAULT_TABLE table=test-table [id=ede8bd76c9354ecf8914f8fe56c202e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:26.675580 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2bfb3d2a3e743f7aae5087994bd0743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.676867 25075 tablet_bootstrap.cc:492] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Bootstrap starting.
I20260619 02:56:26.678404 25100 ts_tablet_manager.cc:1905] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:26.679096 25075 tablet_bootstrap.cc:654] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.682134 25074 tablet_bootstrap.cc:492] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Bootstrap starting.
I20260619 02:56:26.682868 25076 tablet_bootstrap.cc:492] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Bootstrap starting.
I20260619 02:56:26.683730 25074 tablet_bootstrap.cc:654] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.684402 25076 tablet_bootstrap.cc:654] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.686478 25100 ts_tablet_manager.cc:1918] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:26.686756 25100 log.cc:1196] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/34953dd7b3db43018efd86b5918db615
I20260619 02:56:26.687417 25100 ts_tablet_manager.cc:1939] T 34953dd7b3db43018efd86b5918db615 P ec4d090a17754943a92f90e1dd25f791: Deleting consensus metadata
I20260619 02:56:26.690196 25075 tablet_bootstrap.cc:492] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: No bootstrap required, opened a new log
I20260619 02:56:26.690415 25075 ts_tablet_manager.cc:1397] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20260619 02:56:26.690483 24555 catalog_manager.cc:4841] TS ec4d090a17754943a92f90e1dd25f791 (127.22.230.2:39967): tablet 34953dd7b3db43018efd86b5918db615 (table test-table [id=fe8240f421264b829d3ba019587fd9e1]) successfully deleted
I20260619 02:56:26.690809 25098 ts_tablet_manager.cc:1905] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:26.692554 25075 raft_consensus.cc:348] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.692854 25075 raft_consensus.cc:374] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.692989 25075 raft_consensus.cc:729] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4d090a17754943a92f90e1dd25f791, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.693310 25075 consensus_queue.cc:260] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.694545 25074 tablet_bootstrap.cc:492] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: No bootstrap required, opened a new log
I20260619 02:56:26.694746 25074 ts_tablet_manager.cc:1397] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20260619 02:56:26.695878 25076 tablet_bootstrap.cc:492] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: No bootstrap required, opened a new log
I20260619 02:56:26.696022 25076 ts_tablet_manager.cc:1397] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260619 02:56:26.696781 25076 raft_consensus.cc:348] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.696986 25076 raft_consensus.cc:374] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.697084 25076 raft_consensus.cc:729] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbfef5bf5a44825824fa589da3176ce, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.697189 25074 raft_consensus.cc:348] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.697348 25076 consensus_queue.cc:260] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.697894 25076 ts_tablet_manager.cc:1428] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 02:56:26.698055 25099 raft_consensus.cc:2258] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:26.698414 25074 raft_consensus.cc:374] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.698587 25074 raft_consensus.cc:729] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0a59fa929c4d259e2370d754790f74, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.698904 25074 consensus_queue.cc:260] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.700204 25074 ts_tablet_manager.cc:1428] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 02:56:26.700690 25099 ts_tablet_manager.cc:1905] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:26.700727 25075 ts_tablet_manager.cc:1428] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20260619 02:56:26.705606 25098 ts_tablet_manager.cc:1918] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:26.705755 25098 log.cc:1196] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/34953dd7b3db43018efd86b5918db615
I20260619 02:56:26.706282 25098 ts_tablet_manager.cc:1939] T 34953dd7b3db43018efd86b5918db615 P 7d0a59fa929c4d259e2370d754790f74: Deleting consensus metadata
I20260619 02:56:26.707396 25099 ts_tablet_manager.cc:1918] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:26.707542 25099 log.cc:1196] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/34953dd7b3db43018efd86b5918db615
I20260619 02:56:26.708053 25099 ts_tablet_manager.cc:1939] T 34953dd7b3db43018efd86b5918db615 P 2cbfef5bf5a44825824fa589da3176ce: Deleting consensus metadata
I20260619 02:56:26.708860 24556 catalog_manager.cc:4841] TS 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835): tablet 34953dd7b3db43018efd86b5918db615 (table test-table [id=fe8240f421264b829d3ba019587fd9e1]) successfully deleted
I20260619 02:56:26.709597 24557 catalog_manager.cc:4841] TS 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803): tablet 34953dd7b3db43018efd86b5918db615 (table test-table [id=fe8240f421264b829d3ba019587fd9e1]) successfully deleted
I20260619 02:56:26.767885 25093 raft_consensus.cc:482] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:26.768133 25093 raft_consensus.cc:504] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.768953 25093 leader_election.cc:288] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835), 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803)
I20260619 02:56:26.769479 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfb3d2a3e743f7aae5087994bd0743" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0a59fa929c4d259e2370d754790f74" is_pre_election: true
I20260619 02:56:26.769500 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfb3d2a3e743f7aae5087994bd0743" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbfef5bf5a44825824fa589da3176ce" is_pre_election: true
I20260619 02:56:26.769707 24732 raft_consensus.cc:2454] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 0.
I20260619 02:56:26.769775 25010 raft_consensus.cc:2454] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 0.
I20260619 02:56:26.770160 24798 leader_election.cc:302] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0a59fa929c4d259e2370d754790f74, ec4d090a17754943a92f90e1dd25f791; no voters:
I20260619 02:56:26.770469 25093 raft_consensus.cc:2781] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:26.770637 25093 raft_consensus.cc:482] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:26.770763 25093 raft_consensus.cc:3037] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:26.772881 25093 raft_consensus.cc:504] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.773418 25093 leader_election.cc:288] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 election: Requested vote from peers 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835), 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803)
I20260619 02:56:26.773979 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfb3d2a3e743f7aae5087994bd0743" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbfef5bf5a44825824fa589da3176ce"
I20260619 02:56:26.773962 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2bfb3d2a3e743f7aae5087994bd0743" candidate_uuid: "ec4d090a17754943a92f90e1dd25f791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0a59fa929c4d259e2370d754790f74"
I20260619 02:56:26.774163 24732 raft_consensus.cc:3037] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:26.774174 25010 raft_consensus.cc:3037] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:26.777029 24732 raft_consensus.cc:2454] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 1.
I20260619 02:56:26.777029 25010 raft_consensus.cc:2454] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4d090a17754943a92f90e1dd25f791 in term 1.
I20260619 02:56:26.777424 24798 leader_election.cc:302] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0a59fa929c4d259e2370d754790f74, ec4d090a17754943a92f90e1dd25f791; no voters:
I20260619 02:56:26.777725 25093 raft_consensus.cc:2781] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:26.777962 25093 raft_consensus.cc:686] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 1 LEADER]: Becoming Leader. State: Replica: ec4d090a17754943a92f90e1dd25f791, State: Running, Role: LEADER
I20260619 02:56:26.778299 25093 consensus_queue.cc:237] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.780890 24571 catalog_manager.cc:5495] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4d090a17754943a92f90e1dd25f791 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "ec4d090a17754943a92f90e1dd25f791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:56:26 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837784, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:26.850946 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.833225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34229 (local address 127.0.0.1:51720)
0619 02:56:26.834345 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:26.834364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:26.834499 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:26.834902 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:26.834907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:26.835718 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:26.836077 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.836089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.837557 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.837562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.839066 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:26.839078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.840130 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.840137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:26.840313 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.841052 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:26.841102 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:26.843913 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:26.848097 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:26.848109 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:26.848118 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:26.848666 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:26.849083 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:26.849088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:26.849092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:26.849305 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:26.849901 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:26.849910 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:26.850088 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 02:56:26.850265 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 02:56:26.850521 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":411,"threads_started":1}
I20260619 02:56:26.851426 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:26.833102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.22.230.62:34229)
0619 02:56:26.833285 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:26.833291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:26.834463 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 02:56:26.834634 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:26.834640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:26.834752 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:26.834853 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:26.836248 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.837428 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.839209 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:26.839946 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:26.840068 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:26.844103 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:26.844109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:26.844114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:26.844154 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:26.847967 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.848785 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.848790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.848794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.848968 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:26.849440 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:26.849445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:26.849450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:26.849774 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:26.849911 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:26.850377 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 02:56:26.850718 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":72}
I20260619 02:56:26.856266 24571 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:51720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:56:26.857360 24571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:26.864008 24706 tablet_service.cc:1467] Processing CreateTablet for tablet 8df05b4456934f11a5cfa779b9af8952 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21031e0a82d74cb1b05b54c23c9bd84b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:56:26.864212 24845 tablet_service.cc:1467] Processing CreateTablet for tablet 8df05b4456934f11a5cfa779b9af8952 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21031e0a82d74cb1b05b54c23c9bd84b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:56:26.864353 24985 tablet_service.cc:1467] Processing CreateTablet for tablet 8df05b4456934f11a5cfa779b9af8952 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=21031e0a82d74cb1b05b54c23c9bd84b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:56:26.864535 24706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df05b4456934f11a5cfa779b9af8952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.864729 24845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df05b4456934f11a5cfa779b9af8952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.864837 24985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df05b4456934f11a5cfa779b9af8952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:26.871596 25076 tablet_bootstrap.cc:492] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: Bootstrap starting.
I20260619 02:56:26.871738 25074 tablet_bootstrap.cc:492] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: Bootstrap starting.
I20260619 02:56:26.872977 25075 tablet_bootstrap.cc:492] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: Bootstrap starting.
I20260619 02:56:26.873632 25074 tablet_bootstrap.cc:654] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.873632 25076 tablet_bootstrap.cc:654] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.874735 25075 tablet_bootstrap.cc:654] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:26.884548 25074 tablet_bootstrap.cc:492] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: No bootstrap required, opened a new log
I20260619 02:56:26.884682 25076 tablet_bootstrap.cc:492] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: No bootstrap required, opened a new log
I20260619 02:56:26.884861 25076 ts_tablet_manager.cc:1397] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.007s
I20260619 02:56:26.884862 25074 ts_tablet_manager.cc:1397] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260619 02:56:26.885597 25074 raft_consensus.cc:348] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.885597 25076 raft_consensus.cc:348] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.885838 25076 raft_consensus.cc:374] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.885919 25076 raft_consensus.cc:729] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbfef5bf5a44825824fa589da3176ce, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.886081 25074 raft_consensus.cc:374] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.886224 25074 raft_consensus.cc:729] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0a59fa929c4d259e2370d754790f74, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.886230 25076 consensus_queue.cc:260] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.886595 25074 consensus_queue.cc:260] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.886834 25076 ts_tablet_manager.cc:1428] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:56:26.886705 25082 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:26.887037 25082 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:26.887277 25074 ts_tablet_manager.cc:1428] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:56:26.887898 25081 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:26.888236 25081 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:26.889369 25075 tablet_bootstrap.cc:492] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: No bootstrap required, opened a new log
I20260619 02:56:26.889537 25075 ts_tablet_manager.cc:1397] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.011s
I20260619 02:56:26.890305 25075 raft_consensus.cc:348] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.890537 25075 raft_consensus.cc:374] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:26.890635 25075 raft_consensus.cc:729] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4d090a17754943a92f90e1dd25f791, State: Initialized, Role: FOLLOWER
I20260619 02:56:26.890859 25075 consensus_queue.cc:260] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:26.891374 25093 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:26.891644 25093 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:26.892079 25075 ts_tablet_manager.cc:1428] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:56:27.153915 25081 raft_consensus.cc:482] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:27.154109 25081 raft_consensus.cc:504] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:27.155606 25081 leader_election.cc:288] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec4d090a17754943a92f90e1dd25f791 (127.22.230.2:39967), 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803)
I20260619 02:56:27.174948 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.156031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:36637 (local address 127.22.230.3:40803)
0619 02:56:27.168312 (+ 12281us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.168318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.168338 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:56:27.168428 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.168433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.168508 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.168633 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:27.169076 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.170129 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.172724 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.173928 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.174085 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.174159 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.174486 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.174659 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12177,"thread_start_us":12075,"threads_started":1}
I20260619 02:56:27.174934 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.155949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:51457 (local address 127.22.230.2:39967)
0619 02:56:27.167957 (+ 12008us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.167964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.168444 (+ 480us) server_negotiation.cc:407] Connection header received
0619 02:56:27.168519 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.168523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.168626 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.168712 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:27.169488 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.170408 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.172770 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.173927 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.174085 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.174182 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.174485 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.174647 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11881,"thread_start_us":11967,"threads_started":1}
I20260619 02:56:27.175122 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.156094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:39967 (local address 127.22.230.1:51457)
0619 02:56:27.168348 (+ 12254us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:27.168361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:27.168444 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:27.169074 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:27.169076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:27.169091 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:27.169350 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.169356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.170527 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.170533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.172634 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.172646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.174124 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.174129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.174300 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.174350 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 02:56:27.174444 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:56:27.174624 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12183,"mutex_wait_us":94,"thread_start_us":11359,"threads_started":1}
I20260619 02:56:27.175653 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.156080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40803 (local address 127.22.230.1:36637)
0619 02:56:27.167755 (+ 11675us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:27.167797 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:27.167968 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:27.168668 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:27.168674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:27.168703 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:27.168945 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.168953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.170262 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.170265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.172538 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.172558 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.174101 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.174113 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.174294 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.174360 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 02:56:27.174444 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 02:56:27.174624 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11511,"mutex_wait_us":61,"thread_start_us":12104,"threads_started":1}
I20260619 02:56:27.176030 24870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df05b4456934f11a5cfa779b9af8952" candidate_uuid: "7d0a59fa929c4d259e2370d754790f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec4d090a17754943a92f90e1dd25f791" is_pre_election: true
I20260619 02:56:27.176270 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df05b4456934f11a5cfa779b9af8952" candidate_uuid: "7d0a59fa929c4d259e2370d754790f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbfef5bf5a44825824fa589da3176ce" is_pre_election: true
I20260619 02:56:27.176400 24870 raft_consensus.cc:2454] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0a59fa929c4d259e2370d754790f74 in term 0.
I20260619 02:56:27.176492 25010 raft_consensus.cc:2454] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0a59fa929c4d259e2370d754790f74 in term 0.
I20260619 02:56:27.176929 24658 leader_election.cc:302] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0a59fa929c4d259e2370d754790f74, ec4d090a17754943a92f90e1dd25f791; no voters:
I20260619 02:56:27.177261 25081 raft_consensus.cc:2781] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:27.177374 25081 raft_consensus.cc:482] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:27.177505 25081 raft_consensus.cc:3037] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:27.179630 25081 raft_consensus.cc:504] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:27.180284 25081 leader_election.cc:288] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [CANDIDATE]: Term 1 election: Requested vote from peers ec4d090a17754943a92f90e1dd25f791 (127.22.230.2:39967), 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803)
I20260619 02:56:27.180743 24870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df05b4456934f11a5cfa779b9af8952" candidate_uuid: "7d0a59fa929c4d259e2370d754790f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec4d090a17754943a92f90e1dd25f791"
I20260619 02:56:27.180747 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df05b4456934f11a5cfa779b9af8952" candidate_uuid: "7d0a59fa929c4d259e2370d754790f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbfef5bf5a44825824fa589da3176ce"
I20260619 02:56:27.180915 24870 raft_consensus.cc:3037] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:27.180945 25010 raft_consensus.cc:3037] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:27.183017 24870 raft_consensus.cc:2454] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0a59fa929c4d259e2370d754790f74 in term 1.
I20260619 02:56:27.183116 25010 raft_consensus.cc:2454] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0a59fa929c4d259e2370d754790f74 in term 1.
I20260619 02:56:27.183418 24658 leader_election.cc:302] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0a59fa929c4d259e2370d754790f74, ec4d090a17754943a92f90e1dd25f791; no voters:
I20260619 02:56:27.183714 25081 raft_consensus.cc:2781] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:27.184100 25081 raft_consensus.cc:686] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [term 1 LEADER]: Becoming Leader. State: Replica: 7d0a59fa929c4d259e2370d754790f74, State: Running, Role: LEADER
I20260619 02:56:27.184507 25081 consensus_queue.cc:237] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } }
I20260619 02:56:27.186928 25118 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: TxnStatusTablet state changed. Reason: New leader 7d0a59fa929c4d259e2370d754790f74. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.187381 25118 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:56:27.188160 25120 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:56:27.188362 25120 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:56:27.188647 24571 catalog_manager.cc:5495] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260619 02:56:27.244827 25093 consensus_queue.cc:1035] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:27.259156 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.245942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40835 (local address 127.0.0.1:36432)
0619 02:56:27.246145 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:27.246165 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:27.246293 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:27.249582 (+ 3289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:27.249590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:27.249611 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:27.249888 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.249902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.252341 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.252350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.254491 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.254512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.255869 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.255878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.256161 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.256263 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:27.257951 (+ 1688us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:27.257980 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:56:27.258061 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:56:27.258276 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 02:56:27.260548 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.246050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.22.230.1:40835)
0619 02:56:27.249016 (+ 2966us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.249029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.249054 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:56:27.249170 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.249178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.249308 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.250058 (+ 750us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:27.250464 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.252145 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.254746 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.255644 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.256487 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.256694 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:27.257788 (+ 1094us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:27.259913 (+ 2125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.259993 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.260109 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2804,"thread_start_us":2771,"threads_started":1}
I20260619 02:56:27.261024 25125 consensus_queue.cc:1035] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:27.275731 25010 raft_consensus.cc:1261] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Refusing update from remote peer 7d0a59fa929c4d259e2370d754790f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:27.275971 24870 raft_consensus.cc:1261] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Refusing update from remote peer 7d0a59fa929c4d259e2370d754790f74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:27.276662 25081 consensus_queue.cc:1035] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:27.277707 25081 consensus_queue.cc:1035] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 02:56:27.281170 25093 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: TxnStatusTablet state changed. Reason: New leader 7d0a59fa929c4d259e2370d754790f74. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.281523 25093 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:27.285163 25093 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.285410 25093 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P ec4d090a17754943a92f90e1dd25f791: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:27.291229 25082 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: TxnStatusTablet state changed. Reason: New leader 7d0a59fa929c4d259e2370d754790f74. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.291484 25082 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:27.292488 25118 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.292721 25118 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:56:27.299326 25081 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.301769 25081 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 7d0a59fa929c4d259e2370d754790f74: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:56:27.302394 25082 tablet_replica.cc:439] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d0a59fa929c4d259e2370d754790f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0a59fa929c4d259e2370d754790f74" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40835 } } peers { permanent_uuid: "ec4d090a17754943a92f90e1dd25f791" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 39967 } } peers { permanent_uuid: "2cbfef5bf5a44825824fa589da3176ce" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40803 } } }
I20260619 02:56:27.302667 25082 tablet_replica.cc:442] T 8df05b4456934f11a5cfa779b9af8952 P 2cbfef5bf5a44825824fa589da3176ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:27.329679 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.316391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:41605 (local address 127.22.230.1:40835)
0619 02:56:27.316554 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.316560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.323153 (+ 6593us) server_negotiation.cc:407] Connection header received
0619 02:56:27.323330 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.323335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.323415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.323509 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:27.324104 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.325134 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.327016 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.327700 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.327791 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.328844 (+ 1053us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:27.328967 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:27.329023 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.329262 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.329342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:56:27.330621 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.316256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40835 (local address 127.22.230.2:41605)
0619 02:56:27.323034 (+ 6778us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:27.323054 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:27.323202 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:27.323540 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:27.323545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:27.323558 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:27.323946 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.323973 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.325287 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.325291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.326873 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.326887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.328396 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.328402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.328583 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.328645 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:27.329095 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:27.329111 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 02:56:27.330109 (+ 998us) client_negotiation.cc:240] Negotiation successful
0619 02:56:27.330243 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6607,"thread_start_us":6491,"threads_started":1}
I20260619 02:56:27.378073 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.369958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:36123 (local address 127.22.230.2:39967)
0619 02:56:27.370127 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.370131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.372337 (+ 2206us) server_negotiation.cc:407] Connection header received
0619 02:56:27.372519 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.372524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.372612 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.372701 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:27.373403 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.374281 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.375733 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.376330 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.376447 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.377383 (+ 936us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:27.377540 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:27.377640 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.377843 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.377900 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:56:27.378346 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.369572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:39967 (local address 127.22.230.1:36123)
0619 02:56:27.372223 (+ 2651us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:27.372247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:27.372385 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:27.372869 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:27.372874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:27.372888 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:27.373259 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.373267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.374401 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.374407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.375596 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:27.375607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.376519 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.376525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:27.376732 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.376795 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:27.377718 (+ 923us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:27.377727 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 02:56:27.377808 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 02:56:27.377947 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2442,"thread_start_us":2276,"threads_started":1}
I20260619 02:56:27.408339 24571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:27.408604 24571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:27.411754 24571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16df71b4970e48eab5ab472972af276c: Sending DeleteTablet for 3 replicas of tablet a2bfb3d2a3e743f7aae5087994bd0743
I20260619 02:56:27.412487 24706 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bfb3d2a3e743f7aae5087994bd0743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:27 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:33844
I20260619 02:56:27.412653 24985 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bfb3d2a3e743f7aae5087994bd0743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:27 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56930
I20260619 02:56:27.412664 24845 tablet_service.cc:1514] Processing DeleteTablet for tablet a2bfb3d2a3e743f7aae5087994bd0743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:27 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:58940
I20260619 02:56:27.424506 25143 tablet_replica.cc:330] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: stopping tablet replica
I20260619 02:56:27.425459 25143 raft_consensus.cc:2229] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:27.426234 25143 raft_consensus.cc:2258] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:27.428014 25143 ts_tablet_manager.cc:1905] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:27.429019 25145 tablet_replica.cc:330] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: stopping tablet replica
I20260619 02:56:27.429237 25145 raft_consensus.cc:2229] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:27.429555 25145 raft_consensus.cc:2258] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:27.430877 25145 ts_tablet_manager.cc:1905] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:27.431991 25144 tablet_replica.cc:330] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: stopping tablet replica
I20260619 02:56:27.432422 25144 raft_consensus.cc:2229] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:27.432946 25144 raft_consensus.cc:2258] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:27.435599 25144 ts_tablet_manager.cc:1905] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:27.436213 25143 ts_tablet_manager.cc:1918] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:27.436448 25143 log.cc:1196] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/a2bfb3d2a3e743f7aae5087994bd0743
I20260619 02:56:27.436962 25143 ts_tablet_manager.cc:1939] T a2bfb3d2a3e743f7aae5087994bd0743 P 7d0a59fa929c4d259e2370d754790f74: Deleting consensus metadata
I20260619 02:56:27.437902 25145 ts_tablet_manager.cc:1918] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:27.438097 25145 log.cc:1196] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/a2bfb3d2a3e743f7aae5087994bd0743
I20260619 02:56:27.438088 24556 catalog_manager.cc:4841] TS 7d0a59fa929c4d259e2370d754790f74 (127.22.230.1:40835): tablet a2bfb3d2a3e743f7aae5087994bd0743 (table test-table [id=ede8bd76c9354ecf8914f8fe56c202e5]) successfully deleted
I20260619 02:56:27.438684 25145 ts_tablet_manager.cc:1939] T a2bfb3d2a3e743f7aae5087994bd0743 P ec4d090a17754943a92f90e1dd25f791: Deleting consensus metadata
I20260619 02:56:27.439675 24555 catalog_manager.cc:4841] TS ec4d090a17754943a92f90e1dd25f791 (127.22.230.2:39967): tablet a2bfb3d2a3e743f7aae5087994bd0743 (table test-table [id=ede8bd76c9354ecf8914f8fe56c202e5]) successfully deleted
I20260619 02:56:27.444432 25144 ts_tablet_manager.cc:1918] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:27.444559 25144 log.cc:1196] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/a2bfb3d2a3e743f7aae5087994bd0743
Jun 19 02:56:27 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:27.445000 25144 ts_tablet_manager.cc:1939] T a2bfb3d2a3e743f7aae5087994bd0743 P 2cbfef5bf5a44825824fa589da3176ce: Deleting consensus metadata
I20260619 02:56:27.446019 24557 catalog_manager.cc:4841] TS 2cbfef5bf5a44825824fa589da3176ce (127.22.230.3:40803): tablet a2bfb3d2a3e743f7aae5087994bd0743 (table test-table [id=ede8bd76c9354ecf8914f8fe56c202e5]) successfully deleted
I20260619 02:56:27.454248 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.427422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.22.230.1:40835)
0619 02:56:27.427679 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.427689 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.431671 (+ 3982us) server_negotiation.cc:407] Connection header received
0619 02:56:27.433170 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.433176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.433261 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.433343 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:27.438195 (+ 4852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.439924 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.441140 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.441616 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.441779 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.445959 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:27.445973 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:27.445992 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:27.446029 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:27.451192 (+ 5163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.451928 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.451941 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.451945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.452107 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.452523 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.452528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.452532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.452955 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:27.453268 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.453612 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.453846 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 02:56:27.455047 24772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36444
I20260619 02:56:27.472185 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.457881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.22.230.62:34229)
0619 02:56:27.458490 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.458496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.458514 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:56:27.458619 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.458627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.458705 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.458814 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:27.460092 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.461310 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.462861 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.463483 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.463613 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.466204 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:27.466210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:27.466215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:27.466253 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:27.469580 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.470148 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.470153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.470156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.470352 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.470735 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.470741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.470745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.471059 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:27.471332 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.471579 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.471792 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":353,"threads_started":1}
W20260619 02:56:27.472748 24571 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51728
I20260619 02:56:27.474529 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24642
I20260619 02:56:27.488734 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.475291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.22.230.1:40835)
0619 02:56:27.475459 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.475464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.476389 (+ 925us) server_negotiation.cc:407] Connection header received
0619 02:56:27.476495 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.476500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.476590 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.476678 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:27.477696 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.478717 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.479868 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.480529 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.480651 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.483427 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:27.483431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:27.483434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:27.483463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:27.486235 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.486983 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.486987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.486989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.487106 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.487463 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.487466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.487468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.487746 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:27.487989 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.488240 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.488402 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:56:27.489277 24772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36458
W20260619 02:56:27.489851 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24642: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:27.490042 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24642
I20260619 02:56:27.509706 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24781
Jun 19 02:56:27 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:27.523090 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.510361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.22.230.2:39967)
0619 02:56:27.510561 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.510589 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.510612 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:56:27.510862 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.510869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.510952 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.511033 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:27.512092 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.512808 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.513906 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.514304 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.514388 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.518072 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:27.518084 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:27.518118 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:27.518164 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:27.520784 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.521354 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.521360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.521363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.521487 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.521771 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.521774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.521776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.522092 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:27.522303 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.522509 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.522740 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 02:56:27.523742 24911 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062
W20260619 02:56:27.524333 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24781: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:27.524467 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24781
I20260619 02:56:27.545320 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24920
Jun 19 02:56:27 dist-test-slave-2gms krb5kdc[24526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:27.559257 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:27.546044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.22.230.3:40803)
0619 02:56:27.546281 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:27.546286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:27.546302 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:56:27.546351 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:27.546354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:27.546412 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:27.546494 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:27.547680 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.548370 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.549527 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:27.549933 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:27.550020 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:27.554079 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:27.554092 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:27.554107 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:27.554154 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:27.556865 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.557416 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.557423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.557426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.557554 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:27.557937 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:27.557940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:27.557942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:27.558263 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:27.558470 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:27.558759 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:56:27.558916 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 02:56:27.559911 25050 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39152
W20260619 02:56:27.560604 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:27.560789 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24920
I20260619 02:56:27.578008 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 24539
I20260619 02:56:27.579100 24611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:27.692396 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 24539
2026-06-19T02:56:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4643 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:27 dist-test-slave-2gms krb5kdc[25155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:27 dist-test-slave-2gms krb5kdc[25155](info): set up 2 sockets
Jun 19 02:56:27 dist-test-slave-2gms krb5kdc[25155](info): commencing operation
krb5kdc: starting...
W20260619 02:56:28.774552 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:28 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:28.951969 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:43369
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:29.257656 25168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:29.257923 25168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:29.258010 25168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:29.264973 25168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:29.265065 25168 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:56:29.265105 25168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:29.265168 25168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:29.265218 25168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:29.265269 25168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:29.274644 25168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:43369
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:29.275729 25168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:29.276891 25168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:29.288007 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:29.288002 25176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:29.288038 25174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:29.288749 25168 server_base.cc:1034] running on GCE node
I20260619 02:56:29.289463 25168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:29.290736 25168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:29.291944 25168 hybrid_clock.cc:648] HybridClock initialized: now 1781837789291892 us; error 25 us; skew 500 ppm
I20260619 02:56:29.292251 25168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:29 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:29.295315 25168 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:29.296633 25168 webserver.cc:460] Webserver started at http://127.22.230.62:42027/ using document root <none> and password file <none>
I20260619 02:56:29.297336 25168 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:29.297577 25168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:29.297894 25168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:29.300652 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "7c1032cc00544ab38b083da776b531be"
format_stamp: "Formatted at 2026-06-19 02:56:29 on dist-test-slave-2gms"
server_key: "a6adf90f730472f6de362b12f769c0f1"
server_key_iv: "2b2891b3bb9178367f91953ab9be745d"
server_key_version: "encryptionkey@0"
I20260619 02:56:29.301306 25168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "7c1032cc00544ab38b083da776b531be"
format_stamp: "Formatted at 2026-06-19 02:56:29 on dist-test-slave-2gms"
server_key: "a6adf90f730472f6de362b12f769c0f1"
server_key_iv: "2b2891b3bb9178367f91953ab9be745d"
server_key_version: "encryptionkey@0"
I20260619 02:56:29.305961 25168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:56:29.309144 25183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:29.310369 25168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:56:29.310539 25168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "7c1032cc00544ab38b083da776b531be"
format_stamp: "Formatted at 2026-06-19 02:56:29 on dist-test-slave-2gms"
server_key: "a6adf90f730472f6de362b12f769c0f1"
server_key_iv: "2b2891b3bb9178367f91953ab9be745d"
server_key_version: "encryptionkey@0"
I20260619 02:56:29.310817 25168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:29.330139 25168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:29.333709 25168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:29.333971 25168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:29.354364 25168 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:43369
I20260619 02:56:29.354396 25244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:43369 every 8 connection(s)
I20260619 02:56:29.355659 25168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:29.357949 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 25168
I20260619 02:56:29.358142 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:29.358879 23448 external_mini_cluster.cc:1428] Setting key 8c87d325592e58dcf41c0138dd43eadb
I20260619 02:56:29.359439 25245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:29.372900 25245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Bootstrap starting.
Jun 19 02:56:29 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:29.376497 25245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:29.377766 25245 log.cc:824] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:29.380995 25245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: No bootstrap required, opened a new log
I20260619 02:56:29.382442 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:29.360711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.22.230.62:43369)
0619 02:56:29.361457 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:29.361471 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:29.361518 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:56:29.363948 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:29.363973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:29.364440 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:29.364887 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:29.366271 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.368253 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.369458 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.370596 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.370784 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:29.374449 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:29.374459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:29.374477 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:29.374522 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:29.378272 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:29.379154 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:29.379162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:29.379169 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:29.379358 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:29.379677 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:29.379682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:29.379684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:29.380267 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:29.380551 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:29.380834 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 02:56:29.381210 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":299,"threads_started":1}
I20260619 02:56:29.388859 25245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1032cc00544ab38b083da776b531be" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43369 } }
I20260619 02:56:29.389209 25245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:29.389312 25245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1032cc00544ab38b083da776b531be, State: Initialized, Role: FOLLOWER
I20260619 02:56:29.389963 25245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1032cc00544ab38b083da776b531be" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43369 } }
I20260619 02:56:29.390204 25245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:29.390352 25245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:29.390508 25245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:29.392530 25245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1032cc00544ab38b083da776b531be" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43369 } }
I20260619 02:56:29.392987 25245 leader_election.cc:302] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c1032cc00544ab38b083da776b531be; no voters:
I20260619 02:56:29.393503 25245 leader_election.cc:288] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:29.393635 25250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:29.394469 25250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [term 1 LEADER]: Becoming Leader. State: Replica: 7c1032cc00544ab38b083da776b531be, State: Running, Role: LEADER
I20260619 02:56:29.395076 25250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1032cc00544ab38b083da776b531be" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43369 } }
I20260619 02:56:29.395726 25245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:29.397805 25251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c1032cc00544ab38b083da776b531be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1032cc00544ab38b083da776b531be" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43369 } } }
I20260619 02:56:29.397984 25252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c1032cc00544ab38b083da776b531be. Latest consensus state: current_term: 1 leader_uuid: "7c1032cc00544ab38b083da776b531be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1032cc00544ab38b083da776b531be" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43369 } } }
I20260619 02:56:29.398176 25251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:29.398232 25252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:29.399768 25258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:29.405288 25258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:29.415191 25258 catalog_manager.cc:1347] Generated new cluster ID: dbf01a31dd2a4f929ad5344f907035f6
I20260619 02:56:29.415298 25258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:29.431649 25258 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:29.432904 25258 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:29.443835 25258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Generated new TSK 0
I20260619 02:56:29.444599 25258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:29.489748 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43369
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:29.802356 25271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:29.802699 25271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:29.802878 25271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:29.810232 25271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:29.810328 25271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:29.810508 25271 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:56:29.810607 25271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:56:29.810727 25271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:29.821175 25271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:29.822400 25271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:29.823717 25271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:29.836444 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:29.836539 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:29.836418 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:29.837306 25271 server_base.cc:1034] running on GCE node
I20260619 02:56:29.837944 25271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:29.838761 25271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:29.839977 25271 hybrid_clock.cc:648] HybridClock initialized: now 1781837789839938 us; error 31 us; skew 500 ppm
I20260619 02:56:29.840322 25271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:29 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:29.843254 25271 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:29.844615 25271 webserver.cc:460] Webserver started at http://127.22.230.1:36357/ using document root <none> and password file <none>
I20260619 02:56:29.845248 25271 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:29.845391 25271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:29.845687 25271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:29.848464 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
format_stamp: "Formatted at 2026-06-19 02:56:29 on dist-test-slave-2gms"
server_key: "f017cff2586a47cec2cced56f26f9407"
server_key_iv: "3be32e33cf8b931f03c37546d2fc3d5b"
server_key_version: "encryptionkey@0"
I20260619 02:56:29.849162 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
format_stamp: "Formatted at 2026-06-19 02:56:29 on dist-test-slave-2gms"
server_key: "f017cff2586a47cec2cced56f26f9407"
server_key_iv: "3be32e33cf8b931f03c37546d2fc3d5b"
server_key_version: "encryptionkey@0"
I20260619 02:56:29.853770 25271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:29.856999 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:29.858172 25271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:56:29.858326 25271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
format_stamp: "Formatted at 2026-06-19 02:56:29 on dist-test-slave-2gms"
server_key: "f017cff2586a47cec2cced56f26f9407"
server_key_iv: "3be32e33cf8b931f03c37546d2fc3d5b"
server_key_version: "encryptionkey@0"
I20260619 02:56:29.858525 25271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:29.884781 25271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:29.888490 25271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:29.888772 25271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:29.891223 25271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:29.891371 25271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:29.891507 25271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:29.891623 25271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:56:29 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:29.916663 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:29.891717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:39523 (local address 127.22.230.62:43369)
0619 02:56:29.892471 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:29.892477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:29.893471 (+ 994us) server_negotiation.cc:407] Connection header received
0619 02:56:29.896641 (+ 3170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:29.896649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:29.896751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:29.896875 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:29.899721 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.900690 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.903898 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.904403 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.904544 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:29.909395 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:29.909403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:29.909410 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:29.909456 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:29.912700 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:29.913746 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:29.913753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:29.913758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:29.913908 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:29.914468 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:29.914476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:29.914482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:29.914965 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:29.915165 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:29.915832 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 02:56:29.916122 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":527,"threads_started":1}
I20260619 02:56:29.917088 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:29.892400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.22.230.1:39523)
0619 02:56:29.893291 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:29.893348 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:29.896178 (+ 2830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:29.897368 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:29.897382 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:29.898402 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:29.899523 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:29.899557 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.900910 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.900923 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:29.903725 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:29.903744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.904688 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.904698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:29.905036 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:29.905884 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:29.905925 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:29.909133 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:29.912837 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:29.912846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:29.912864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:29.913557 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:29.914012 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:29.914017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:29.914020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:29.914313 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:29.915103 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:29.915108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:29.915448 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 02:56:29.915657 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 02:56:29.916004 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":343,"threads_started":1}
I20260619 02:56:29.943244 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:29.928941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:51127 (local address 127.22.230.62:43369)
0619 02:56:29.929150 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:29.929156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:29.930132 (+ 976us) server_negotiation.cc:407] Connection header received
0619 02:56:29.930261 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:29.930269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:29.930364 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:29.930479 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:29.931356 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.932748 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.934037 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.934943 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.935079 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:29.937629 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:29.937635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:29.937640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:29.937679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:29.940752 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:29.941373 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:29.941380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:29.941384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:29.941496 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:29.941828 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:29.941835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:29.941838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:29.942084 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:29.942243 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:29.942643 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:56:29.942860 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:56:29.943344 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:29.929365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.22.230.1:51127)
0619 02:56:29.930049 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:29.930065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:29.930168 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:29.930507 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:29.930510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:29.930951 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:29.931221 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:29.931229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.932885 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.932888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:29.933874 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:29.933885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:29.935327 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:29.935337 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:29.935637 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:29.936450 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:29.936467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:29.937453 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:29.940889 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:29.940894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:29.940898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:29.941253 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:29.941603 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:29.941608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:29.941611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:29.941728 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:29.942199 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:29.942204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:29.942383 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 02:56:29.942524 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 02:56:29.942863 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":404,"threads_started":1}
I20260619 02:56:29.951359 25271 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:43967
I20260619 02:56:29.951401 25406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:43967 every 8 connection(s)
I20260619 02:56:29.952507 25271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:29.954414 25407 heartbeater.cc:347] Connected to a master server at 127.22.230.62:43369
I20260619 02:56:29.954658 25407 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:29.955230 25407 heartbeater.cc:511] Master 127.22.230.62:43369 requested a full tablet report, sending...
I20260619 02:56:29.956990 25200 ts_manager.cc:194] Registered new tserver with Master: 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967)
I20260619 02:56:29.958499 25200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:39523
I20260619 02:56:29.958730 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 25271
I20260619 02:56:29.958926 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:29.959720 23448 external_mini_cluster.cc:1428] Setting key da3de5d872406de4e8e6c77cd845be2d
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:29.997880 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43369
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:30.308149 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:30.308410 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:30.308512 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:30.315553 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:30.315671 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:30.315824 25411 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:56:30.315900 25411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:56:30.316039 25411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:30.325012 25411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:30.326231 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:30.327553 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:30.339632 25416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:30.339663 25419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:30.339594 25417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:30.340523 25411 server_base.cc:1034] running on GCE node
I20260619 02:56:30.341185 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:30.342000 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:30.343225 25411 hybrid_clock.cc:648] HybridClock initialized: now 1781837790343195 us; error 40 us; skew 500 ppm
I20260619 02:56:30.343511 25411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:30 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:30.346378 25411 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:30.347865 25411 webserver.cc:460] Webserver started at http://127.22.230.2:44319/ using document root <none> and password file <none>
I20260619 02:56:30.348492 25411 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:30.348645 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:30.348933 25411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:30.351686 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "eed7cc539e09416986e0121bff049340"
format_stamp: "Formatted at 2026-06-19 02:56:30 on dist-test-slave-2gms"
server_key: "8fa8868ca7826a36f9314c51661ccc8a"
server_key_iv: "733c29ccba7bfb13f9715fdef31baa8c"
server_key_version: "encryptionkey@0"
I20260619 02:56:30.352417 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "eed7cc539e09416986e0121bff049340"
format_stamp: "Formatted at 2026-06-19 02:56:30 on dist-test-slave-2gms"
server_key: "8fa8868ca7826a36f9314c51661ccc8a"
server_key_iv: "733c29ccba7bfb13f9715fdef31baa8c"
server_key_version: "encryptionkey@0"
I20260619 02:56:30.358174 25411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 02:56:30.362356 25426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:30.363940 25411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:30.364084 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "eed7cc539e09416986e0121bff049340"
format_stamp: "Formatted at 2026-06-19 02:56:30 on dist-test-slave-2gms"
server_key: "8fa8868ca7826a36f9314c51661ccc8a"
server_key_iv: "733c29ccba7bfb13f9715fdef31baa8c"
server_key_version: "encryptionkey@0"
I20260619 02:56:30.364275 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:30.396747 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:30.400602 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:30.400830 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:30.403353 25411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:30.403440 25411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:30.403666 25411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:30.403769 25411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:56:30 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:30.428130 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.403804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:45303 (local address 127.22.230.62:43369)
0619 02:56:30.404848 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:30.404853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:30.405523 (+ 670us) server_negotiation.cc:407] Connection header received
0619 02:56:30.408581 (+ 3058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:30.408586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:30.408674 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:30.408791 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:30.411005 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.412303 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.414698 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.415189 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.415333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.420527 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:30.420532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:30.420535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:30.420565 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:30.424665 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.425571 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.425575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.425577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.425698 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.426289 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.426294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.426298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.426731 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:30.426883 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:30.427591 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 02:56:30.427796 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942}
I20260619 02:56:30.429103 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.404483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.22.230.2:45303)
0619 02:56:30.405340 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:30.405396 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:30.408169 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:30.409121 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:30.409135 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:30.409929 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:30.410789 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.410835 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.412478 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.412485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.414522 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.414533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.415447 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.415454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.415708 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.416549 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:30.416596 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:30.420248 (+ 3652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:30.424819 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.424826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.424836 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.425401 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.425873 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.425880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.425885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.426159 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.426875 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:30.426882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:30.427228 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 02:56:30.427484 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 02:56:30.427977 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":341,"threads_started":1}
I20260619 02:56:30.448313 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.434556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.22.230.2:46785)
0619 02:56:30.435270 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:30.435295 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:30.435454 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:30.435802 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:30.435810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:30.436529 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:30.436997 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.437013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.438188 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.438193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.439776 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.439789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.440536 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.440541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.440721 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.441376 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:30.441391 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:30.442031 (+ 640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:30.445195 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.445214 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.445218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.445791 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.446162 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.446168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.446173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.446347 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.447178 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:30.447182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:30.447436 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 02:56:30.447623 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 02:56:30.448004 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":350,"threads_started":1}
I20260619 02:56:30.448802 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.434489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:46785 (local address 127.22.230.62:43369)
0619 02:56:30.434707 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:30.434715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:30.435390 (+ 675us) server_negotiation.cc:407] Connection header received
0619 02:56:30.435596 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:30.435600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:30.435663 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:30.435740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:30.437139 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.438038 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.439957 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.440350 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.440496 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.442202 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:30.442206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:30.442209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:30.442236 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:30.445072 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.445929 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.445936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.445941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.446057 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.446504 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.446509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.446513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.446953 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:30.447121 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:30.448056 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 02:56:30.448348 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:56:30.458487 25411 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:35547
I20260619 02:56:30.458526 25545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:35547 every 8 connection(s)
I20260619 02:56:30.460016 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:30.462333 25546 heartbeater.cc:347] Connected to a master server at 127.22.230.62:43369
I20260619 02:56:30.462687 25546 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:30.463316 25546 heartbeater.cc:511] Master 127.22.230.62:43369 requested a full tablet report, sending...
I20260619 02:56:30.464557 25200 ts_manager.cc:194] Registered new tserver with Master: eed7cc539e09416986e0121bff049340 (127.22.230.2:35547)
I20260619 02:56:30.464602 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 25411
I20260619 02:56:30.464787 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:30.465206 23448 external_mini_cluster.cc:1428] Setting key a582aca68da8401cd31b667b4c36e6a0
I20260619 02:56:30.465461 25200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:45303
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:30.502681 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43369
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:30.806671 25550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:30.806921 25550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:30.807092 25550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:30.813951 25550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:30.814044 25550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:30.814216 25550 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 02:56:30.814273 25550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:56:30.814405 25550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:30.823836 25550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:30.825121 25550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:30.826867 25550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:30.839483 25555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:30.839520 25556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:30.839541 25558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:30.840411 25550 server_base.cc:1034] running on GCE node
I20260619 02:56:30.840989 25550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:30.841866 25550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:30.843056 25550 hybrid_clock.cc:648] HybridClock initialized: now 1781837790843017 us; error 30 us; skew 500 ppm
I20260619 02:56:30.843377 25550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:30 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:30.849522 25550 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:30.850929 25550 webserver.cc:460] Webserver started at http://127.22.230.3:34845/ using document root <none> and password file <none>
I20260619 02:56:30.851521 25550 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:30.851613 25550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:30.851850 25550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:30.854509 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "99026503292a48328ae1c2e8c76903bd"
format_stamp: "Formatted at 2026-06-19 02:56:30 on dist-test-slave-2gms"
server_key: "008bd79a727b2ff7f9fd27f7c8a1ae15"
server_key_iv: "a6e241db17ab53badf518c311dd715cd"
server_key_version: "encryptionkey@0"
I20260619 02:56:30.855249 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "99026503292a48328ae1c2e8c76903bd"
format_stamp: "Formatted at 2026-06-19 02:56:30 on dist-test-slave-2gms"
server_key: "008bd79a727b2ff7f9fd27f7c8a1ae15"
server_key_iv: "a6e241db17ab53badf518c311dd715cd"
server_key_version: "encryptionkey@0"
I20260619 02:56:30.859862 25550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:56:30.863289 25565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:30.864508 25550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:56:30.864660 25550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "99026503292a48328ae1c2e8c76903bd"
format_stamp: "Formatted at 2026-06-19 02:56:30 on dist-test-slave-2gms"
server_key: "008bd79a727b2ff7f9fd27f7c8a1ae15"
server_key_iv: "a6e241db17ab53badf518c311dd715cd"
server_key_version: "encryptionkey@0"
I20260619 02:56:30.864887 25550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:30.880764 25550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:30.884799 25550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:30.885082 25550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:30.887602 25550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:30.887749 25550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:30.887878 25550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:30.888005 25550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 02:56:30 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837790, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:30.912626 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.888024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:55123 (local address 127.22.230.62:43369)
0619 02:56:30.888215 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:30.888225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:30.889703 (+ 1478us) server_negotiation.cc:407] Connection header received
0619 02:56:30.891719 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:30.891727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:30.891823 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:30.891935 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:30.894350 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.895735 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.898721 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.899200 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.899302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.904832 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:30.904841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:30.904848 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:30.904897 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:30.908199 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.909367 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.909373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.909376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.909542 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.910172 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.910180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.910186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.910617 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:30.910786 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:30.911944 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0619 02:56:30.912200 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:56:30.914463 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.888649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.22.230.3:55123)
0619 02:56:30.889509 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:30.889561 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:30.891450 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:30.892168 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:30.892183 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:30.892923 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:30.894063 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.894081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.895945 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.895953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.898524 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.898543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.899475 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.899485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.899819 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.900682 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:30.900721 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:30.904561 (+ 3840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:30.908349 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.908361 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.908385 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.909182 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.909702 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.909713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.909716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.910013 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.910868 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:30.910878 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:30.911414 (+ 536us) client_negotiation.cc:769] Sending connection context
0619 02:56:30.911786 (+ 372us) client_negotiation.cc:240] Negotiation successful
0619 02:56:30.912465 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":364,"threads_started":1}
I20260619 02:56:30.936900 25550 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40751
I20260619 02:56:30.936921 25684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40751 every 8 connection(s)
I20260619 02:56:30.938401 25550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:30.939338 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 25550
I20260619 02:56:30.939507 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:30.940059 23448 external_mini_cluster.cc:1428] Setting key 2aa1fdb0585105ddd3d70ddde28b843f
I20260619 02:56:30.940639 25685 heartbeater.cc:347] Connected to a master server at 127.22.230.62:43369
I20260619 02:56:30.940883 25685 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:30.941754 25685 heartbeater.cc:511] Master 127.22.230.62:43369 requested a full tablet report, sending...
I20260619 02:56:30.943297 25199 ts_manager.cc:194] Registered new tserver with Master: 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:30.944243 25199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:55123
I20260619 02:56:30.947331 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.926317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.22.230.3:42389)
0619 02:56:30.931128 (+ 4811us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:30.931203 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:30.931412 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:30.931961 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:30.931973 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:30.932772 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:30.933392 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.933412 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.935145 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.935150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.936944 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:30.936959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.937726 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.937732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:30.937932 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.938523 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:30.938545 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:30.939902 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:30.944835 (+ 4933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.944842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.944846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.945384 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.945805 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:30.945808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:30.945811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:30.945925 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:30.946456 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:30.946459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:30.946660 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 02:56:30.946772 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:56:30.946992 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4653,"thread_start_us":4508,"threads_started":1}
I20260619 02:56:30.947455 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.925878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:42389 (local address 127.22.230.62:43369)
0619 02:56:30.926093 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:30.926099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:30.931307 (+ 5208us) server_negotiation.cc:407] Connection header received
0619 02:56:30.931556 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:30.931564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:30.931651 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:30.931758 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:30.933604 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.934928 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.937104 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.937540 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.937638 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.940152 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:30.940162 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:30.940170 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:30.940218 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:30.944726 (+ 4508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.945536 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.945542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.945546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.945692 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.946040 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.946044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.946047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.946343 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:30.946485 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:30.946898 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 02:56:30.947125 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:56:30.955631 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:56:30.961252 25407 heartbeater.cc:503] Master 127.22.230.62:43369 was elected leader, sending a full tablet report...
Jun 19 02:56:30 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:56:30 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:31.000713 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:30.988059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.22.230.62:43369)
0619 02:56:30.988238 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:30.988242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:30.988566 (+ 324us) server_negotiation.cc:407] Connection header received
0619 02:56:30.988704 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:30.988709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:30.988770 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:30.988853 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:30.990123 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.990914 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.992050 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:30.992438 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:30.992526 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:30.996060 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:30.996065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:30.996068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:30.996100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:30.998632 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.999191 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.999194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.999197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.999300 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:30.999607 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:30.999610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:30.999612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:30.999893 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:31.000054 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.000315 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.000459 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:56:31.004447 25199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:31.011051 25199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:31.044271 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.022413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35547 (local address 127.0.0.1:53082)
0619 02:56:31.023319 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.023360 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.023543 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.036893 (+ 13350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.036904 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.036946 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:31.037461 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.037477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.039554 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.039559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.041431 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.041443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.043259 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.043268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.043464 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.043520 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.043639 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.043827 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":40,"thread_start_us":449,"threads_started":1}
I20260619 02:56:31.044327 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.023079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40751 (local address 127.0.0.1:60230)
0619 02:56:31.035282 (+ 12203us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.035307 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.035460 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.038655 (+ 3195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.038658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.038682 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:31.039035 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.039044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.040780 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.040785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.042170 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.042180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.043717 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.043724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.043894 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.043917 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.043972 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.044050 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12042,"thread_start_us":11685,"threads_started":1}
I20260619 02:56:31.045234 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.022690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.22.230.2:35547)
0619 02:56:31.036072 (+ 13382us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.036085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.036121 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:56:31.036243 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.036254 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.036585 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.037710 (+ 1125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:31.037930 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.039403 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.041587 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.043012 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.044504 (+ 1492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.044706 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.044784 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.044948 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13157,"thread_start_us":13007,"threads_started":1}
I20260619 02:56:31.045439 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.023631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.22.230.3:40751)
0619 02:56:31.038190 (+ 14559us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.038196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.038219 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:56:31.038291 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.038296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.038458 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.038662 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:31.039211 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.040641 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.042323 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.043497 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.044785 (+ 1288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.044993 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.045043 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.045149 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14381,"thread_start_us":14202,"threads_started":1}
I20260619 02:56:31.046051 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.024108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.22.230.1:43967)
0619 02:56:31.036925 (+ 12817us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.036933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.036961 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:56:31.037048 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.037054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.037311 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.037503 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:31.038150 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.039580 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.043494 (+ 3914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.044925 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.045025 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.045211 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.045740 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.045859 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12665,"thread_start_us":12787,"threads_started":1}
I20260619 02:56:31.046118 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.024433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:43967 (local address 127.0.0.1:40878)
0619 02:56:31.036328 (+ 11895us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.036345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.036473 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.037494 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.037498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.037527 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:31.037928 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.037939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.039736 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.039740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.043326 (+ 3586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.043338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.045389 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.045396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.045569 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.045599 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.045691 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.045798 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11773,"mutex_wait_us":27,"thread_start_us":11658,"threads_started":1}
I20260619 02:56:31.049038 25476 tablet_service.cc:1467] Processing CreateTablet for tablet b2576d4b0dfa46aeab72c14a125721d3 (DEFAULT_TABLE table=test-table [id=479169f8e6514c5a9f832bc894ef60ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:31.049332 25615 tablet_service.cc:1467] Processing CreateTablet for tablet b2576d4b0dfa46aeab72c14a125721d3 (DEFAULT_TABLE table=test-table [id=479169f8e6514c5a9f832bc894ef60ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:31.049433 25337 tablet_service.cc:1467] Processing CreateTablet for tablet b2576d4b0dfa46aeab72c14a125721d3 (DEFAULT_TABLE table=test-table [id=479169f8e6514c5a9f832bc894ef60ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:31.050475 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2576d4b0dfa46aeab72c14a125721d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.050520 25615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2576d4b0dfa46aeab72c14a125721d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.050743 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2576d4b0dfa46aeab72c14a125721d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.073820 25704 tablet_bootstrap.cc:492] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Bootstrap starting.
I20260619 02:56:31.077904 25704 tablet_bootstrap.cc:654] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.079337 25704 log.cc:824] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:31.080575 25705 tablet_bootstrap.cc:492] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Bootstrap starting.
I20260619 02:56:31.084631 25705 tablet_bootstrap.cc:654] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.085906 25705 log.cc:824] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:31.087416 25706 tablet_bootstrap.cc:492] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Bootstrap starting.
I20260619 02:56:31.089607 25704 tablet_bootstrap.cc:492] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: No bootstrap required, opened a new log
I20260619 02:56:31.090080 25704 ts_tablet_manager.cc:1397] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260619 02:56:31.090410 25706 tablet_bootstrap.cc:654] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.091639 25706 log.cc:824] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:31.092998 25705 tablet_bootstrap.cc:492] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: No bootstrap required, opened a new log
I20260619 02:56:31.093366 25705 ts_tablet_manager.cc:1397] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260619 02:56:31.104091 25704 raft_consensus.cc:348] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.104579 25704 raft_consensus.cc:374] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.104684 25704 raft_consensus.cc:729] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff7a256f9b44f8fb73cdd2a6afd7479, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.105140 25706 tablet_bootstrap.cc:492] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: No bootstrap required, opened a new log
I20260619 02:56:31.105407 25706 ts_tablet_manager.cc:1397] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260619 02:56:31.105433 25704 consensus_queue.cc:260] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.106112 25705 raft_consensus.cc:348] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.106554 25705 raft_consensus.cc:374] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.106726 25705 raft_consensus.cc:729] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed7cc539e09416986e0121bff049340, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.107349 25705 consensus_queue.cc:260] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.119949 25704 ts_tablet_manager.cc:1428] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent starting tablet: real 0.030s user 0.017s sys 0.011s
I20260619 02:56:31.120944 25546 heartbeater.cc:503] Master 127.22.230.62:43369 was elected leader, sending a full tablet report...
I20260619 02:56:31.121616 25705 ts_tablet_manager.cc:1428] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Time spent starting tablet: real 0.028s user 0.027s sys 0.000s
I20260619 02:56:31.122408 25706 raft_consensus.cc:348] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.122960 25706 raft_consensus.cc:374] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.123100 25706 raft_consensus.cc:729] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99026503292a48328ae1c2e8c76903bd, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.123757 25706 consensus_queue.cc:260] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.126368 25706 ts_tablet_manager.cc:1428] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Time spent starting tablet: real 0.021s user 0.019s sys 0.000s
I20260619 02:56:31.126775 25685 heartbeater.cc:503] Master 127.22.230.62:43369 was elected leader, sending a full tablet report...
W20260619 02:56:31.195434 25686 tablet.cc:2369] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:56:31.204658 25408 tablet.cc:2369] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:56:31.210832 25711 raft_consensus.cc:482] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:31.211158 25711 raft_consensus.cc:504] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
W20260619 02:56:31.212479 25547 tablet.cc:2369] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:56:31.213454 25711 leader_election.cc:288] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:31.222018 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.214110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:39203 (local address 127.22.230.1:43967)
0619 02:56:31.214349 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.214355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.215306 (+ 951us) server_negotiation.cc:407] Connection header received
0619 02:56:31.215657 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.215665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.215767 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.215854 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:31.216493 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.217831 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.220359 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.221114 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.221293 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.221370 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.221645 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.221776 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:56:31.222159 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.213852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:43967 (local address 127.22.230.2:39203)
0619 02:56:31.215189 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.215211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.215473 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.215889 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.215893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.215916 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:31.216330 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.216341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.217973 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.217978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.220217 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.220227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.221325 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.221331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.221490 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.221513 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.221621 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.221743 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":48,"thread_start_us":920,"threads_started":1}
I20260619 02:56:31.223403 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.213896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57497 (local address 127.22.230.3:40751)
0619 02:56:31.214120 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.214128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.215716 (+ 1588us) server_negotiation.cc:407] Connection header received
0619 02:56:31.215877 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.215885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.215971 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.216068 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:31.216682 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.217407 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.220089 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.221667 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.221865 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.221963 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.223031 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.223157 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:56:31.223191 25357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2576d4b0dfa46aeab72c14a125721d3" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" is_pre_election: true
I20260619 02:56:31.223717 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.214373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40751 (local address 127.22.230.2:57497)
0619 02:56:31.215604 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.215621 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.215744 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.216106 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.216109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.216129 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:31.216530 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.216539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.217554 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.217561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.219907 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.219920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.222406 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.222416 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.222810 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.222864 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.222974 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.223136 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":4,"thread_start_us":858,"threads_started":1}
I20260619 02:56:31.223760 25357 raft_consensus.cc:2454] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:31.224428 25430 leader_election.cc:302] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff7a256f9b44f8fb73cdd2a6afd7479, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:31.224807 25711 raft_consensus.cc:2781] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:31.224944 25711 raft_consensus.cc:482] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:31.224778 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2576d4b0dfa46aeab72c14a125721d3" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd" is_pre_election: true
I20260619 02:56:31.225044 25711 raft_consensus.cc:3037] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:31.225363 25635 raft_consensus.cc:2454] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:31.227501 25711 raft_consensus.cc:504] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.228204 25711 leader_election.cc:288] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:31.228540 25357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2576d4b0dfa46aeab72c14a125721d3" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
I20260619 02:56:31.228721 25357 raft_consensus.cc:3037] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:31.228704 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2576d4b0dfa46aeab72c14a125721d3" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd"
I20260619 02:56:31.228920 25635 raft_consensus.cc:3037] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:31.230952 25357 raft_consensus.cc:2454] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:31.231289 25430 leader_election.cc:302] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff7a256f9b44f8fb73cdd2a6afd7479, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:31.231557 25711 raft_consensus.cc:2781] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:31.232107 25711 raft_consensus.cc:686] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Becoming Leader. State: Replica: eed7cc539e09416986e0121bff049340, State: Running, Role: LEADER
I20260619 02:56:31.232613 25711 consensus_queue.cc:237] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.232839 25635 raft_consensus.cc:2454] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:31.237808 25200 catalog_manager.cc:5495] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed7cc539e09416986e0121bff049340 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:56:31 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837789, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:31.348613 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.333284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.22.230.62:43369)
0619 02:56:31.333496 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.333509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.334049 (+ 540us) server_negotiation.cc:407] Connection header received
0619 02:56:31.334194 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.334202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.334311 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.334435 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:31.335516 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.336877 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.338133 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.339035 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.339208 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.343125 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:31.343130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:31.343134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:31.343160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:31.345618 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:31.346641 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:31.346646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:31.346651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:31.346790 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:31.347196 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:31.347199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:31.347201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:31.347439 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:31.347540 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.348005 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.348309 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":56}
I20260619 02:56:31.348692 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.333262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43369 (local address 127.0.0.1:54606)
0619 02:56:31.333941 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.333960 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.334074 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.334459 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.334462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.335127 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:31.335377 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.335385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.337020 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.337024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.337982 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.337993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.339265 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.339270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.339443 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.340079 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:31.340095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:31.342942 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:31.345773 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:31.345790 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:31.345801 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:31.346451 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:31.346901 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:31.346905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:31.346908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:31.347092 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:31.347559 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:31.347568 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:31.347754 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.347930 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.348187 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":350,"threads_started":1}
I20260619 02:56:31.353339 25200 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:54606:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:56:31.354403 25200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:31.361129 25337 tablet_service.cc:1467] Processing CreateTablet for tablet 64d83c0ac85c4cc5934996326825c768 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c9f871ce3f845f796f0b44a6e56d84b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:56:31.361129 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 64d83c0ac85c4cc5934996326825c768 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c9f871ce3f845f796f0b44a6e56d84b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:56:31.362104 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d83c0ac85c4cc5934996326825c768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.362100 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d83c0ac85c4cc5934996326825c768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.363034 25615 tablet_service.cc:1467] Processing CreateTablet for tablet 64d83c0ac85c4cc5934996326825c768 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c9f871ce3f845f796f0b44a6e56d84b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:56:31.363848 25615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d83c0ac85c4cc5934996326825c768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.370119 25705 tablet_bootstrap.cc:492] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: Bootstrap starting.
I20260619 02:56:31.371758 25705 tablet_bootstrap.cc:654] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.372278 25706 tablet_bootstrap.cc:492] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: Bootstrap starting.
I20260619 02:56:31.373769 25706 tablet_bootstrap.cc:654] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.374609 25704 tablet_bootstrap.cc:492] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Bootstrap starting.
I20260619 02:56:31.375878 25706 tablet_bootstrap.cc:492] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: No bootstrap required, opened a new log
I20260619 02:56:31.376001 25706 ts_tablet_manager.cc:1397] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 02:56:31.376166 25704 tablet_bootstrap.cc:654] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.376502 25705 tablet_bootstrap.cc:492] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: No bootstrap required, opened a new log
I20260619 02:56:31.376659 25705 ts_tablet_manager.cc:1397] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 02:56:31.376787 25706 raft_consensus.cc:348] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.377000 25706 raft_consensus.cc:374] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.377089 25706 raft_consensus.cc:729] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99026503292a48328ae1c2e8c76903bd, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.377367 25706 consensus_queue.cc:260] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.377460 25705 raft_consensus.cc:348] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.377734 25705 raft_consensus.cc:374] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.377825 25705 raft_consensus.cc:729] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed7cc539e09416986e0121bff049340, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.378016 25706 ts_tablet_manager.cc:1428] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:56:31.377874 25712 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.378047 25705 consensus_queue.cc:260] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.378234 25712 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.378440 25711 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.378729 25711 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.379266 25705 ts_tablet_manager.cc:1428] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 02:56:31.379659 25704 tablet_bootstrap.cc:492] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: No bootstrap required, opened a new log
I20260619 02:56:31.379786 25704 ts_tablet_manager.cc:1397] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:56:31.380374 25704 raft_consensus.cc:348] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.380604 25704 raft_consensus.cc:374] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.380707 25704 raft_consensus.cc:729] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff7a256f9b44f8fb73cdd2a6afd7479, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.380915 25704 consensus_queue.cc:260] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.381358 25710 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.381735 25710 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.382190 25704 ts_tablet_manager.cc:1428] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:56:31.555719 25711 raft_consensus.cc:482] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:31.555956 25711 raft_consensus.cc:504] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.556614 25711 leader_election.cc:288] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:31.557066 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d83c0ac85c4cc5934996326825c768" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd" is_pre_election: true
I20260619 02:56:31.557071 25357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d83c0ac85c4cc5934996326825c768" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" is_pre_election: true
I20260619 02:56:31.557281 25635 raft_consensus.cc:2454] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:31.557303 25357 raft_consensus.cc:2454] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:31.557677 25427 leader_election.cc:302] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99026503292a48328ae1c2e8c76903bd, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:31.557948 25711 raft_consensus.cc:2781] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:31.558023 25711 raft_consensus.cc:482] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:31.558087 25711 raft_consensus.cc:3037] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:31.559914 25711 raft_consensus.cc:504] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.560429 25711 leader_election.cc:288] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:31.560752 25357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d83c0ac85c4cc5934996326825c768" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
I20260619 02:56:31.560837 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d83c0ac85c4cc5934996326825c768" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd"
I20260619 02:56:31.560976 25357 raft_consensus.cc:3037] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:31.561008 25635 raft_consensus.cc:3037] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:31.563818 25635 raft_consensus.cc:2454] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:31.563818 25357 raft_consensus.cc:2454] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:31.564177 25427 leader_election.cc:302] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99026503292a48328ae1c2e8c76903bd, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:31.564445 25711 raft_consensus.cc:2781] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:31.564617 25711 raft_consensus.cc:686] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Becoming Leader. State: Replica: eed7cc539e09416986e0121bff049340, State: Running, Role: LEADER
I20260619 02:56:31.564857 25711 consensus_queue.cc:237] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.566385 25715 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: TxnStatusTablet state changed. Reason: New leader eed7cc539e09416986e0121bff049340. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.566663 25715 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:56:31.567286 25727 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:56:31.567553 25727 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:56:31.567579 25200 catalog_manager.cc:5495] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed7cc539e09416986e0121bff049340 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
I20260619 02:56:31.615375 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.609148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35547 (local address 127.0.0.1:53084)
0619 02:56:31.609381 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.609408 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.609582 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.610511 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.610515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.610525 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:31.610743 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.610751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.611799 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.611807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.612763 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.612773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.613541 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.613547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.613740 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.613789 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:31.614903 (+ 1114us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:31.614934 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.615059 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.615173 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:56:31.615701 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.609211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.22.230.2:35547)
0619 02:56:31.610178 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.610183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.610196 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:56:31.610263 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.610266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.610334 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.610494 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:31.610894 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.611682 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.612904 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.613381 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.613515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.614027 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:31.614806 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:31.614951 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.615097 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.615175 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":713,"threads_started":1}
I20260619 02:56:31.634730 25357 raft_consensus.cc:1261] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:31.634862 25635 raft_consensus.cc:1261] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:31.635648 25715 consensus_queue.cc:1035] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:31.636268 25711 consensus_queue.cc:1035] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:31.643493 25710 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: TxnStatusTablet state changed. Reason: New leader eed7cc539e09416986e0121bff049340. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.643784 25710 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.650918 25731 mvcc.cc:205] Tried to move back new op lower bound from 7298407594526732288 to 7298407594251689984. Current Snapshot: MvccSnapshot[applied={T|T < 7298407594526732288}]
I20260619 02:56:31.651822 25715 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.652033 25715 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:56:31.653656 25710 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.653880 25710 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P 5ff7a256f9b44f8fb73cdd2a6afd7479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.653822 25712 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: TxnStatusTablet state changed. Reason: New leader eed7cc539e09416986e0121bff049340. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.654045 25712 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.656430 25715 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.656672 25715 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P eed7cc539e09416986e0121bff049340: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:56:31.658556 25712 tablet_replica.cc:439] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } } }
I20260619 02:56:31.658798 25712 tablet_replica.cc:442] T 64d83c0ac85c4cc5934996326825c768 P 99026503292a48328ae1c2e8c76903bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:56:31.666636 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.660338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.22.230.2:35547)
0619 02:56:31.660536 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.660542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.661260 (+ 718us) server_negotiation.cc:407] Connection header received
0619 02:56:31.661492 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.661497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.661599 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.661689 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:31.662097 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.663293 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.664719 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.665376 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.665495 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.665852 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:31.665972 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:31.666062 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.666205 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.666280 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:56:31.682858 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.676738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57621 (local address 127.22.230.2:35547)
0619 02:56:31.676911 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:31.676917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:31.677547 (+ 630us) server_negotiation.cc:407] Connection header received
0619 02:56:31.677670 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:31.677674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:31.677770 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:31.677859 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:31.678321 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.679495 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.680904 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.681553 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.681684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.682079 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:31.682190 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:31.682283 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:31.682385 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 02:56:31.682451 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:56:31.682880 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:31.676762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35547 (local address 127.22.230.2:57621)
0619 02:56:31.677453 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:31.677468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:31.677571 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:31.677873 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:31.677876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:31.677883 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:31.678195 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.678202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.679653 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.679657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.680766 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:31.680777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:31.681731 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:31.681735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:31.681862 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:31.681905 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:31.682280 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:31.682287 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:56:31.682343 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 02:56:31.682431 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":403,"threads_started":1}
I20260619 02:56:31.691594 25635 raft_consensus.cc:1261] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:31.691640 25357 raft_consensus.cc:1261] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:31.692060 25715 consensus_queue.cc:1035] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:31.692319 25711 consensus_queue.cc:1035] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:31.746858 25200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:31.747129 25200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:31.750231 25200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Sending DeleteTablet for 3 replicas of tablet b2576d4b0dfa46aeab72c14a125721d3
I20260619 02:56:31.751170 25476 tablet_service.cc:1514] Processing DeleteTablet for tablet b2576d4b0dfa46aeab72c14a125721d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:31 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260619 02:56:31.751176 25337 tablet_service.cc:1514] Processing DeleteTablet for tablet b2576d4b0dfa46aeab72c14a125721d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:31 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:40878
I20260619 02:56:31.752135 25615 tablet_service.cc:1514] Processing DeleteTablet for tablet b2576d4b0dfa46aeab72c14a125721d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:31 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:60230
I20260619 02:56:31.756907 25200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:31.757728 25200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:31.762841 25749 tablet_replica.cc:330] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: stopping tablet replica
I20260619 02:56:31.763239 25749 raft_consensus.cc:2229] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:31.763535 25749 raft_consensus.cc:2258] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:31.764219 25748 tablet_replica.cc:330] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: stopping tablet replica
I20260619 02:56:31.764542 25748 raft_consensus.cc:2229] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:31.765100 25748 raft_consensus.cc:2258] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:31.765509 25614 tablet_service.cc:1467] Processing CreateTablet for tablet 24e61eaf73544d0db10e91d20b94ec63 (DEFAULT_TABLE table=test-table [id=1a5b174353404bdbb18a16ef69dcaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:31.765962 25614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e61eaf73544d0db10e91d20b94ec63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.766858 25748 ts_tablet_manager.cc:1905] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:31.771183 25750 tablet_replica.cc:330] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: stopping tablet replica
I20260619 02:56:31.771556 25337 tablet_service.cc:1467] Processing CreateTablet for tablet 24e61eaf73544d0db10e91d20b94ec63 (DEFAULT_TABLE table=test-table [id=1a5b174353404bdbb18a16ef69dcaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:31.771651 25750 raft_consensus.cc:2229] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:31.771764 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 24e61eaf73544d0db10e91d20b94ec63 (DEFAULT_TABLE table=test-table [id=1a5b174353404bdbb18a16ef69dcaa23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:31.771970 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e61eaf73544d0db10e91d20b94ec63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.772157 25750 raft_consensus.cc:2258] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:31.772190 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e61eaf73544d0db10e91d20b94ec63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:31.773808 25750 ts_tablet_manager.cc:1905] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:31.778294 25704 tablet_bootstrap.cc:492] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Bootstrap starting.
I20260619 02:56:31.779861 25704 tablet_bootstrap.cc:654] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.780042 25706 tablet_bootstrap.cc:492] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Bootstrap starting.
I20260619 02:56:31.780843 25750 ts_tablet_manager.cc:1918] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:31.781075 25750 log.cc:1196] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/b2576d4b0dfa46aeab72c14a125721d3
I20260619 02:56:31.781634 25706 tablet_bootstrap.cc:654] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.781759 25750 ts_tablet_manager.cc:1939] T b2576d4b0dfa46aeab72c14a125721d3 P 99026503292a48328ae1c2e8c76903bd: Deleting consensus metadata
I20260619 02:56:31.782002 25705 tablet_bootstrap.cc:492] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Bootstrap starting.
I20260619 02:56:31.782435 25749 ts_tablet_manager.cc:1905] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:31.783550 25705 tablet_bootstrap.cc:654] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:31.784032 25184 catalog_manager.cc:4841] TS 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751): tablet b2576d4b0dfa46aeab72c14a125721d3 (table test-table [id=479169f8e6514c5a9f832bc894ef60ab]) successfully deleted
I20260619 02:56:31.785612 25706 tablet_bootstrap.cc:492] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: No bootstrap required, opened a new log
I20260619 02:56:31.785763 25706 ts_tablet_manager.cc:1397] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260619 02:56:31.786444 25706 raft_consensus.cc:348] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.786688 25706 raft_consensus.cc:374] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.786764 25706 raft_consensus.cc:729] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99026503292a48328ae1c2e8c76903bd, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.787047 25706 consensus_queue.cc:260] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.789667 25706 ts_tablet_manager.cc:1428] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:56:31.790184 25749 ts_tablet_manager.cc:1918] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:31.790381 25749 log.cc:1196] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/b2576d4b0dfa46aeab72c14a125721d3
I20260619 02:56:31.795019 25748 ts_tablet_manager.cc:1918] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:31.795207 25748 log.cc:1196] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/b2576d4b0dfa46aeab72c14a125721d3
I20260619 02:56:31.795712 25748 ts_tablet_manager.cc:1939] T b2576d4b0dfa46aeab72c14a125721d3 P eed7cc539e09416986e0121bff049340: Deleting consensus metadata
I20260619 02:56:31.797423 25187 catalog_manager.cc:4841] TS eed7cc539e09416986e0121bff049340 (127.22.230.2:35547): tablet b2576d4b0dfa46aeab72c14a125721d3 (table test-table [id=479169f8e6514c5a9f832bc894ef60ab]) successfully deleted
I20260619 02:56:31.797621 25749 ts_tablet_manager.cc:1939] T b2576d4b0dfa46aeab72c14a125721d3 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting consensus metadata
I20260619 02:56:31.799163 25187 catalog_manager.cc:4841] TS 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967): tablet b2576d4b0dfa46aeab72c14a125721d3 (table test-table [id=479169f8e6514c5a9f832bc894ef60ab]) successfully deleted
I20260619 02:56:31.800311 25704 tablet_bootstrap.cc:492] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: No bootstrap required, opened a new log
I20260619 02:56:31.800503 25704 ts_tablet_manager.cc:1397] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.005s
I20260619 02:56:31.800805 25705 tablet_bootstrap.cc:492] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: No bootstrap required, opened a new log
I20260619 02:56:31.800987 25705 ts_tablet_manager.cc:1397] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20260619 02:56:31.801259 25704 raft_consensus.cc:348] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.801487 25704 raft_consensus.cc:374] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.801568 25704 raft_consensus.cc:729] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff7a256f9b44f8fb73cdd2a6afd7479, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.801759 25705 raft_consensus.cc:348] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.801831 25704 consensus_queue.cc:260] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.801990 25705 raft_consensus.cc:374] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:31.802071 25705 raft_consensus.cc:729] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed7cc539e09416986e0121bff049340, State: Initialized, Role: FOLLOWER
I20260619 02:56:31.802292 25705 consensus_queue.cc:260] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:31.803037 25705 ts_tablet_manager.cc:1428] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:56:31.804548 25704 ts_tablet_manager.cc:1428] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 02:56:32.076503 25744 raft_consensus.cc:482] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:32.076773 25744 raft_consensus.cc:504] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:32.077638 25744 leader_election.cc:288] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:32.078102 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e61eaf73544d0db10e91d20b94ec63" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" is_pre_election: true
I20260619 02:56:32.078363 25356 raft_consensus.cc:2454] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:32.078402 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e61eaf73544d0db10e91d20b94ec63" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd" is_pre_election: true
I20260619 02:56:32.078651 25635 raft_consensus.cc:2454] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:32.078812 25430 leader_election.cc:302] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff7a256f9b44f8fb73cdd2a6afd7479, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:32.079080 25744 raft_consensus.cc:2781] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:32.079202 25744 raft_consensus.cc:482] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:32.079308 25744 raft_consensus.cc:3037] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:32.081425 25744 raft_consensus.cc:504] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:32.081962 25744 leader_election.cc:288] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:32.082362 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e61eaf73544d0db10e91d20b94ec63" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
I20260619 02:56:32.082455 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24e61eaf73544d0db10e91d20b94ec63" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd"
I20260619 02:56:32.082537 25356 raft_consensus.cc:3037] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:32.082688 25635 raft_consensus.cc:3037] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:32.084489 25356 raft_consensus.cc:2454] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:32.084899 25430 leader_election.cc:302] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff7a256f9b44f8fb73cdd2a6afd7479, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:32.085194 25744 raft_consensus.cc:2781] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:32.085407 25744 raft_consensus.cc:686] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Becoming Leader. State: Replica: eed7cc539e09416986e0121bff049340, State: Running, Role: LEADER
I20260619 02:56:32.085589 25635 raft_consensus.cc:2454] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:32.085711 25744 consensus_queue.cc:237] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:32.092128 25199 catalog_manager.cc:5495] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed7cc539e09416986e0121bff049340 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
I20260619 02:56:32.167135 25634 raft_consensus.cc:1261] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:32.167740 25744 consensus_queue.cc:1035] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:32.177610 25356 raft_consensus.cc:1261] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:32.178145 25755 consensus_queue.cc:1035] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:32.182587 25732 mvcc.cc:205] Tried to move back new op lower bound from 7298407596709814272 to 7298407596385697792. Current Snapshot: MvccSnapshot[applied={T|T < 7298407596709814272}]
W20260619 02:56:32.196420 25731 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:56:33.771371 25199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:33.771574 25199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:33.775110 25199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Sending DeleteTablet for 3 replicas of tablet 24e61eaf73544d0db10e91d20b94ec63
I20260619 02:56:33.775797 25337 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e61eaf73544d0db10e91d20b94ec63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:33 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:40878
I20260619 02:56:33.775828 25475 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e61eaf73544d0db10e91d20b94ec63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:33 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260619 02:56:33.776167 25614 tablet_service.cc:1514] Processing DeleteTablet for tablet 24e61eaf73544d0db10e91d20b94ec63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:33 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:60230
I20260619 02:56:33.788158 25779 tablet_replica.cc:330] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: stopping tablet replica
I20260619 02:56:33.788537 25779 raft_consensus.cc:2229] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:33.788898 25779 raft_consensus.cc:2258] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:33.790205 25779 ts_tablet_manager.cc:1905] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:33.795782 25778 tablet_replica.cc:330] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: stopping tablet replica
I20260619 02:56:33.796015 25780 tablet_replica.cc:330] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: stopping tablet replica
I20260619 02:56:33.796032 25778 raft_consensus.cc:2229] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:33.796228 25780 raft_consensus.cc:2229] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:33.796526 25780 raft_consensus.cc:2258] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:33.796545 25778 raft_consensus.cc:2258] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:33.797375 25780 ts_tablet_manager.cc:1905] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:33.797425 25778 ts_tablet_manager.cc:1905] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:33.799166 25779 ts_tablet_manager.cc:1918] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:56:33.799361 25779 log.cc:1196] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/24e61eaf73544d0db10e91d20b94ec63
I20260619 02:56:33.799966 25779 ts_tablet_manager.cc:1939] T 24e61eaf73544d0db10e91d20b94ec63 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting consensus metadata
I20260619 02:56:33.801245 25187 catalog_manager.cc:4841] TS 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967): tablet 24e61eaf73544d0db10e91d20b94ec63 (table test-table [id=1a5b174353404bdbb18a16ef69dcaa23]) successfully deleted
I20260619 02:56:33.803951 25780 ts_tablet_manager.cc:1918] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:56:33.803951 25778 ts_tablet_manager.cc:1918] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:56:33.804073 25780 log.cc:1196] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/24e61eaf73544d0db10e91d20b94ec63
I20260619 02:56:33.804081 25778 log.cc:1196] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/24e61eaf73544d0db10e91d20b94ec63
I20260619 02:56:33.804513 25778 ts_tablet_manager.cc:1939] T 24e61eaf73544d0db10e91d20b94ec63 P eed7cc539e09416986e0121bff049340: Deleting consensus metadata
I20260619 02:56:33.804513 25780 ts_tablet_manager.cc:1939] T 24e61eaf73544d0db10e91d20b94ec63 P 99026503292a48328ae1c2e8c76903bd: Deleting consensus metadata
I20260619 02:56:33.805660 25184 catalog_manager.cc:4841] TS 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751): tablet 24e61eaf73544d0db10e91d20b94ec63 (table test-table [id=1a5b174353404bdbb18a16ef69dcaa23]) successfully deleted
I20260619 02:56:33.805727 25187 catalog_manager.cc:4841] TS eed7cc539e09416986e0121bff049340 (127.22.230.2:35547): tablet 24e61eaf73544d0db10e91d20b94ec63 (table test-table [id=1a5b174353404bdbb18a16ef69dcaa23]) successfully deleted
Jun 19 02:56:33 dist-test-slave-2gms krb5kdc[25155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:56:33 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:33.833671 25788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.820926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.22.230.62:43369)
0619 02:56:33.821387 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:33.821391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:33.821509 (+ 118us) server_negotiation.cc:407] Connection header received
0619 02:56:33.821657 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:33.821661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:33.821719 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:33.821802 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:33.823087 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.823860 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.825045 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.825415 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.825514 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.829033 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:33.829038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:33.829041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:33.829068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:33.831692 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:33.832205 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:33.832209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:33.832211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:33.832308 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:33.832579 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:33.832582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:33.832583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:33.832845 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:33.833004 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:33.833226 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 02:56:33.833366 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260619 02:56:33.836828 25199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:33.837417 25199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:33.843470 25337 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcebd9e64b649469922859fea2bc298 (DEFAULT_TABLE table=test-table [id=ca7315210eb54815bc42946baa18eaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:33.843557 25475 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcebd9e64b649469922859fea2bc298 (DEFAULT_TABLE table=test-table [id=ca7315210eb54815bc42946baa18eaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:33.843667 25614 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcebd9e64b649469922859fea2bc298 (DEFAULT_TABLE table=test-table [id=ca7315210eb54815bc42946baa18eaf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:33.843868 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcebd9e64b649469922859fea2bc298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:33.844038 25475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcebd9e64b649469922859fea2bc298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:33.844087 25614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcebd9e64b649469922859fea2bc298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:33.850131 25789 tablet_bootstrap.cc:492] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Bootstrap starting.
I20260619 02:56:33.851121 25791 tablet_bootstrap.cc:492] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Bootstrap starting.
I20260619 02:56:33.851887 25789 tablet_bootstrap.cc:654] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:33.852790 25791 tablet_bootstrap.cc:654] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:33.854915 25790 tablet_bootstrap.cc:492] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Bootstrap starting.
I20260619 02:56:33.855940 25789 tablet_bootstrap.cc:492] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: No bootstrap required, opened a new log
I20260619 02:56:33.856073 25789 ts_tablet_manager.cc:1397] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 02:56:33.856492 25790 tablet_bootstrap.cc:654] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:33.856963 25789 raft_consensus.cc:348] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.857195 25789 raft_consensus.cc:374] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:33.857304 25789 raft_consensus.cc:729] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed7cc539e09416986e0121bff049340, State: Initialized, Role: FOLLOWER
I20260619 02:56:33.857604 25789 consensus_queue.cc:260] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.858884 25791 tablet_bootstrap.cc:492] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: No bootstrap required, opened a new log
I20260619 02:56:33.858937 25789 ts_tablet_manager.cc:1428] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 02:56:33.859053 25791 ts_tablet_manager.cc:1397] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 02:56:33.859867 25791 raft_consensus.cc:348] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.860085 25791 raft_consensus.cc:374] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:33.860206 25791 raft_consensus.cc:729] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99026503292a48328ae1c2e8c76903bd, State: Initialized, Role: FOLLOWER
I20260619 02:56:33.860420 25790 tablet_bootstrap.cc:492] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: No bootstrap required, opened a new log
I20260619 02:56:33.860455 25791 consensus_queue.cc:260] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.860693 25790 ts_tablet_manager.cc:1397] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260619 02:56:33.861860 25790 raft_consensus.cc:348] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.862112 25790 raft_consensus.cc:374] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:33.862222 25790 raft_consensus.cc:729] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff7a256f9b44f8fb73cdd2a6afd7479, State: Initialized, Role: FOLLOWER
I20260619 02:56:33.862594 25790 consensus_queue.cc:260] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.863624 25790 ts_tablet_manager.cc:1428] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 02:56:33.864676 25791 ts_tablet_manager.cc:1428] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 02:56:33.872737 25796 raft_consensus.cc:482] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:33.872982 25796 raft_consensus.cc:504] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.874315 25796 leader_election.cc:288] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed7cc539e09416986e0121bff049340 (127.22.230.2:35547), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:33.893162 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.874529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:41421 (local address 127.22.230.2:35547)
0619 02:56:33.887148 (+ 12619us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:33.887157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:33.887601 (+ 444us) server_negotiation.cc:407] Connection header received
0619 02:56:33.887738 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:33.887743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:33.887817 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:33.887923 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:33.888583 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.889334 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.891161 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.892145 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.892276 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.892324 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:33.892677 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:56:33.892846 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12412,"thread_start_us":12263,"threads_started":1}
I20260619 02:56:33.893178 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.874499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35547 (local address 127.22.230.1:41421)
0619 02:56:33.887472 (+ 12973us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:33.887494 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:33.887618 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:33.887951 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:33.887955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:33.887977 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:33.888392 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.888404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.889482 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.889489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.891025 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.891036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.892329 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.892336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.892507 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.892541 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 02:56:33.892639 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:56:33.892800 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12830,"spinlock_wait_cycles":27520,"thread_start_us":12826,"threads_started":1}
I20260619 02:56:33.893757 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.874815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:32935 (local address 127.22.230.3:40751)
0619 02:56:33.886630 (+ 11815us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:33.886639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:33.887724 (+ 1085us) server_negotiation.cc:407] Connection header received
0619 02:56:33.887856 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:33.887860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:33.887965 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:33.888068 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:33.888674 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.889453 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.891731 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.892649 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.892831 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.892896 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:33.893290 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 02:56:33.893432 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11628,"thread_start_us":11478,"threads_started":1}
I20260619 02:56:33.893833 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.874632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40751 (local address 127.22.230.1:32935)
0619 02:56:33.887608 (+ 12976us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:33.887626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:33.887747 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:33.888106 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:33.888110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:33.888158 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:33.888526 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.888538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.889576 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.889580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.891603 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.891614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.892869 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.892876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.893097 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.893130 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:56:33.893377 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 02:56:33.893492 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12861,"spinlock_wait_cycles":1280,"thread_start_us":12514,"threads_started":1}
I20260619 02:56:33.893994 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcebd9e64b649469922859fea2bc298" candidate_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed7cc539e09416986e0121bff049340" is_pre_election: true
I20260619 02:56:33.894454 25501 raft_consensus.cc:2454] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff7a256f9b44f8fb73cdd2a6afd7479 in term 0.
I20260619 02:56:33.894693 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcebd9e64b649469922859fea2bc298" candidate_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd" is_pre_election: true
I20260619 02:56:33.894975 25635 raft_consensus.cc:2454] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff7a256f9b44f8fb73cdd2a6afd7479 in term 0.
I20260619 02:56:33.895565 25290 leader_election.cc:302] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ff7a256f9b44f8fb73cdd2a6afd7479, 99026503292a48328ae1c2e8c76903bd, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:33.895969 25796 raft_consensus.cc:2781] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:33.896095 25796 raft_consensus.cc:482] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:33.896185 25796 raft_consensus.cc:3037] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:33.898218 25796 raft_consensus.cc:504] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.898870 25796 leader_election.cc:288] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [CANDIDATE]: Term 1 election: Requested vote from peers eed7cc539e09416986e0121bff049340 (127.22.230.2:35547), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:33.899163 25501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcebd9e64b649469922859fea2bc298" candidate_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed7cc539e09416986e0121bff049340"
I20260619 02:56:33.899269 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcebd9e64b649469922859fea2bc298" candidate_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd"
I20260619 02:56:33.899363 25501 raft_consensus.cc:3037] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:33.899467 25635 raft_consensus.cc:3037] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:33.901458 25635 raft_consensus.cc:2454] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff7a256f9b44f8fb73cdd2a6afd7479 in term 1.
I20260619 02:56:33.901484 25501 raft_consensus.cc:2454] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff7a256f9b44f8fb73cdd2a6afd7479 in term 1.
I20260619 02:56:33.901844 25287 leader_election.cc:302] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff7a256f9b44f8fb73cdd2a6afd7479, 99026503292a48328ae1c2e8c76903bd; no voters:
I20260619 02:56:33.902130 25796 raft_consensus.cc:2781] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:33.902586 25796 raft_consensus.cc:686] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff7a256f9b44f8fb73cdd2a6afd7479, State: Running, Role: LEADER
I20260619 02:56:33.902988 25796 consensus_queue.cc:237] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:33.906399 25198 catalog_manager.cc:5495] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
I20260619 02:56:33.940616 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.934038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.22.230.1:43967)
0619 02:56:33.934828 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:33.934833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:33.934851 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:56:33.934923 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:33.934926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:33.934991 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:33.935126 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:33.935453 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.936352 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.938280 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.938748 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.938837 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.939355 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:33.939919 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:33.940028 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:33.940147 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:56:33.940203 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":561,"threads_started":1}
I20260619 02:56:33.956075 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.950619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:38997 (local address 127.22.230.2:35547)
0619 02:56:33.950810 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:33.950816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:33.951484 (+ 668us) server_negotiation.cc:407] Connection header received
0619 02:56:33.951684 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:33.951689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:33.951756 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:33.951842 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:33.952350 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.953080 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.954426 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.954872 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.954991 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.955382 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:33.955467 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:33.955557 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:33.955689 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 02:56:33.955756 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:56:33.956266 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.950628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35547 (local address 127.22.230.1:38997)
0619 02:56:33.951372 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:33.951408 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:33.951557 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:33.951865 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:33.951869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:33.951880 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:33.952205 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.952213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.953225 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.953231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.954290 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.954300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.955048 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.955054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.955167 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.955211 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:33.955557 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:33.955569 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:56:33.955668 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:56:33.955785 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":370,"threads_started":1}
I20260619 02:56:33.966781 25635 raft_consensus.cc:1261] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Refusing update from remote peer 5ff7a256f9b44f8fb73cdd2a6afd7479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:33.967005 25501 raft_consensus.cc:1261] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Refusing update from remote peer 5ff7a256f9b44f8fb73cdd2a6afd7479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:33.967432 25801 consensus_queue.cc:1035] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:33.967859 25796 consensus_queue.cc:1035] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:33.975507 25761 mvcc.cc:205] Tried to move back new op lower bound from 7298407604081881088 to 7298407603829837824. Current Snapshot: MvccSnapshot[applied={T|T < 7298407604081881088}]
I20260619 02:56:33.994657 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.988864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:39111 (local address 127.22.230.1:43967)
0619 02:56:33.989031 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:33.989037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:33.989710 (+ 673us) server_negotiation.cc:407] Connection header received
0619 02:56:33.989905 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:33.989909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:33.989979 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:33.990067 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:33.990514 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.991285 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.993317 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.993689 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.993783 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.994092 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:33.994175 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:33.994244 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:33.994349 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:56:33.994402 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:56:33.994944 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:33.988758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:43967 (local address 127.22.230.2:39111)
0619 02:56:33.989609 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:33.989625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:33.989779 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:33.990080 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:33.990084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:33.990095 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:56:33.990351 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.990359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.991413 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.991418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.993199 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:33.993210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:33.993824 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:33.993828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:33.993940 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:33.993972 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:56:33.994252 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:56:33.994259 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:56:33.994317 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 02:56:33.994433 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":540,"threads_started":1}
I20260619 02:56:34.024523 25199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:34.024773 25199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:34.027485 25199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Sending DeleteTablet for 3 replicas of tablet 4bcebd9e64b649469922859fea2bc298
I20260619 02:56:34.028245 25614 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcebd9e64b649469922859fea2bc298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:34 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:60230
I20260619 02:56:34.028411 25337 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcebd9e64b649469922859fea2bc298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:34 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:40878
I20260619 02:56:34.028674 25780 tablet_replica.cc:330] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: stopping tablet replica
I20260619 02:56:34.028664 25779 tablet_replica.cc:330] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: stopping tablet replica
I20260619 02:56:34.028855 25780 raft_consensus.cc:2229] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:34.028865 25779 raft_consensus.cc:2229] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:34.029023 25780 raft_consensus.cc:2258] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:34.029302 25779 raft_consensus.cc:2258] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:34.029484 25475 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcebd9e64b649469922859fea2bc298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:34 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260619 02:56:34.029403 25199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:34.029966 25199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:34.030244 25780 ts_tablet_manager.cc:1905] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:34.030661 25779 ts_tablet_manager.cc:1905] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:34.032497 25778 tablet_replica.cc:330] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: stopping tablet replica
I20260619 02:56:34.032696 25778 raft_consensus.cc:2229] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:34.032858 25778 raft_consensus.cc:2258] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:34.033875 25778 ts_tablet_manager.cc:1905] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:34.037391 25779 ts_tablet_manager.cc:1918] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:34.037551 25779 log.cc:1196] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/4bcebd9e64b649469922859fea2bc298
I20260619 02:56:34.038537 25614 tablet_service.cc:1467] Processing CreateTablet for tablet ad663a9ddea745bf930845f891a68cdb (DEFAULT_TABLE table=test-table [id=cf99374142dc480d90fa3164c9c561e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:34.038831 25337 tablet_service.cc:1467] Processing CreateTablet for tablet ad663a9ddea745bf930845f891a68cdb (DEFAULT_TABLE table=test-table [id=cf99374142dc480d90fa3164c9c561e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:34.038975 25614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad663a9ddea745bf930845f891a68cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:34.039183 25337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad663a9ddea745bf930845f891a68cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:34.042907 25475 tablet_service.cc:1467] Processing CreateTablet for tablet ad663a9ddea745bf930845f891a68cdb (DEFAULT_TABLE table=test-table [id=cf99374142dc480d90fa3164c9c561e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:34.043277 25475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad663a9ddea745bf930845f891a68cdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:34.045305 25780 ts_tablet_manager.cc:1918] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:34.045477 25780 log.cc:1196] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/4bcebd9e64b649469922859fea2bc298
I20260619 02:56:34.045897 25791 tablet_bootstrap.cc:492] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Bootstrap starting.
I20260619 02:56:34.046828 25779 ts_tablet_manager.cc:1939] T 4bcebd9e64b649469922859fea2bc298 P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting consensus metadata
I20260619 02:56:34.047966 25791 tablet_bootstrap.cc:654] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:34.047916 25187 catalog_manager.cc:4841] TS 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967): tablet 4bcebd9e64b649469922859fea2bc298 (table test-table [id=ca7315210eb54815bc42946baa18eaf9]) successfully deleted
I20260619 02:56:34.049464 25780 ts_tablet_manager.cc:1939] T 4bcebd9e64b649469922859fea2bc298 P 99026503292a48328ae1c2e8c76903bd: Deleting consensus metadata
I20260619 02:56:34.049785 25789 tablet_bootstrap.cc:492] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Bootstrap starting.
I20260619 02:56:34.050722 25184 catalog_manager.cc:4841] TS 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751): tablet 4bcebd9e64b649469922859fea2bc298 (table test-table [id=ca7315210eb54815bc42946baa18eaf9]) successfully deleted
I20260619 02:56:34.051276 25789 tablet_bootstrap.cc:654] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:34.062978 25789 tablet_bootstrap.cc:492] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: No bootstrap required, opened a new log
I20260619 02:56:34.063138 25789 ts_tablet_manager.cc:1397] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260619 02:56:34.063901 25789 raft_consensus.cc:348] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.064167 25789 raft_consensus.cc:374] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:34.064247 25789 raft_consensus.cc:729] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed7cc539e09416986e0121bff049340, State: Initialized, Role: FOLLOWER
I20260619 02:56:34.064513 25789 consensus_queue.cc:260] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.067400 25789 ts_tablet_manager.cc:1428] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:56:34.067634 25790 tablet_bootstrap.cc:492] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Bootstrap starting.
I20260619 02:56:34.069968 25790 tablet_bootstrap.cc:654] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:34.072515 25778 ts_tablet_manager.cc:1918] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:56:34.072672 25778 log.cc:1196] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/4bcebd9e64b649469922859fea2bc298
I20260619 02:56:34.073076 25778 ts_tablet_manager.cc:1939] T 4bcebd9e64b649469922859fea2bc298 P eed7cc539e09416986e0121bff049340: Deleting consensus metadata
I20260619 02:56:34.075280 25187 catalog_manager.cc:4841] TS eed7cc539e09416986e0121bff049340 (127.22.230.2:35547): tablet 4bcebd9e64b649469922859fea2bc298 (table test-table [id=ca7315210eb54815bc42946baa18eaf9]) successfully deleted
I20260619 02:56:34.075459 25791 tablet_bootstrap.cc:492] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: No bootstrap required, opened a new log
I20260619 02:56:34.075666 25791 ts_tablet_manager.cc:1397] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.013s
I20260619 02:56:34.076589 25791 raft_consensus.cc:348] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.077026 25791 raft_consensus.cc:374] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:34.077134 25791 raft_consensus.cc:729] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99026503292a48328ae1c2e8c76903bd, State: Initialized, Role: FOLLOWER
I20260619 02:56:34.077837 25791 consensus_queue.cc:260] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.078636 25791 ts_tablet_manager.cc:1428] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260619 02:56:34.079763 25790 tablet_bootstrap.cc:492] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: No bootstrap required, opened a new log
I20260619 02:56:34.079927 25790 ts_tablet_manager.cc:1397] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260619 02:56:34.080714 25790 raft_consensus.cc:348] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.080999 25790 raft_consensus.cc:374] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:34.081081 25790 raft_consensus.cc:729] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff7a256f9b44f8fb73cdd2a6afd7479, State: Initialized, Role: FOLLOWER
I20260619 02:56:34.081331 25790 consensus_queue.cc:260] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.081847 25790 ts_tablet_manager.cc:1428] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 02:56:34.223920 25760 raft_consensus.cc:482] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:34.224144 25760 raft_consensus.cc:504] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.224848 25760 leader_election.cc:288] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:34.225315 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad663a9ddea745bf930845f891a68cdb" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd" is_pre_election: true
I20260619 02:56:34.225354 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad663a9ddea745bf930845f891a68cdb" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" is_pre_election: true
I20260619 02:56:34.225576 25635 raft_consensus.cc:2454] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:34.225611 25356 raft_consensus.cc:2454] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 0.
I20260619 02:56:34.226022 25427 leader_election.cc:302] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99026503292a48328ae1c2e8c76903bd, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:34.226329 25760 raft_consensus.cc:2781] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:34.226457 25760 raft_consensus.cc:482] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:34.226630 25760 raft_consensus.cc:3037] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:34.228560 25760 raft_consensus.cc:504] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.229060 25760 leader_election.cc:288] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967), 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751)
I20260619 02:56:34.229508 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad663a9ddea745bf930845f891a68cdb" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479"
I20260619 02:56:34.229553 25635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad663a9ddea745bf930845f891a68cdb" candidate_uuid: "eed7cc539e09416986e0121bff049340" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99026503292a48328ae1c2e8c76903bd"
I20260619 02:56:34.229678 25356 raft_consensus.cc:3037] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:34.229759 25635 raft_consensus.cc:3037] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:34.231915 25635 raft_consensus.cc:2454] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:34.232311 25427 leader_election.cc:302] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99026503292a48328ae1c2e8c76903bd, eed7cc539e09416986e0121bff049340; no voters:
I20260619 02:56:34.232432 25356 raft_consensus.cc:2454] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed7cc539e09416986e0121bff049340 in term 1.
I20260619 02:56:34.232570 25760 raft_consensus.cc:2781] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:34.232832 25760 raft_consensus.cc:686] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Becoming Leader. State: Replica: eed7cc539e09416986e0121bff049340, State: Running, Role: LEADER
I20260619 02:56:34.233084 25760 consensus_queue.cc:237] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } }
I20260619 02:56:34.236037 25198 catalog_manager.cc:5495] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed7cc539e09416986e0121bff049340 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "eed7cc539e09416986e0121bff049340" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed7cc539e09416986e0121bff049340" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 } health_report { overall_health: UNKNOWN } } }
I20260619 02:56:34.260910 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:34.255467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.22.230.2:35547)
0619 02:56:34.255612 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:34.255617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:34.255969 (+ 352us) server_negotiation.cc:407] Connection header received
0619 02:56:34.256131 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:34.256136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:34.256215 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:34.256307 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:34.256600 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:34.257765 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:34.259112 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:34.259728 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:34.259860 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:34.260208 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:34.260319 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:34.260416 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:34.260567 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 02:56:34.260639 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:56:34.270027 25356 raft_consensus.cc:1261] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:34.270231 25635 raft_consensus.cc:1261] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Refusing update from remote peer eed7cc539e09416986e0121bff049340: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:34.270677 25769 consensus_queue.cc:1035] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff7a256f9b44f8fb73cdd2a6afd7479" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:34.271263 25773 consensus_queue.cc:1035] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99026503292a48328ae1c2e8c76903bd" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 02:56:34.295017 25758 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 02:56:35.874181 25199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:35.874367 25199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:35.877774 25199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c1032cc00544ab38b083da776b531be: Sending DeleteTablet for 3 replicas of tablet ad663a9ddea745bf930845f891a68cdb
I20260619 02:56:35.878466 25337 tablet_service.cc:1514] Processing DeleteTablet for tablet ad663a9ddea745bf930845f891a68cdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:35 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:40878
I20260619 02:56:35.878582 25476 tablet_service.cc:1514] Processing DeleteTablet for tablet ad663a9ddea745bf930845f891a68cdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:35 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:53082
I20260619 02:56:35.878544 25614 tablet_service.cc:1514] Processing DeleteTablet for tablet ad663a9ddea745bf930845f891a68cdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:35 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:60230
I20260619 02:56:35.879179 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25271
I20260619 02:56:35.890254 25842 tablet_replica.cc:330] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: stopping tablet replica
I20260619 02:56:35.890689 25842 raft_consensus.cc:2229] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:35.891099 25842 raft_consensus.cc:2258] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:35.892345 25842 ts_tablet_manager.cc:1905] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:35.895515 25844 tablet_replica.cc:330] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: stopping tablet replica
I20260619 02:56:35.895695 25843 tablet_replica.cc:330] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: stopping tablet replica
I20260619 02:56:35.895713 25844 raft_consensus.cc:2229] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:35.895892 25843 raft_consensus.cc:2229] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:35.895965 25844 raft_consensus.cc:2258] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:35.896286 25843 raft_consensus.cc:2258] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:35.897012 25844 ts_tablet_manager.cc:1905] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:35.897264 25843 ts_tablet_manager.cc:1905] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:35.900687 25842 ts_tablet_manager.cc:1918] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:56:35.900816 25842 log.cc:1196] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/ad663a9ddea745bf930845f891a68cdb
I20260619 02:56:35.901300 25842 ts_tablet_manager.cc:1939] T ad663a9ddea745bf930845f891a68cdb P 5ff7a256f9b44f8fb73cdd2a6afd7479: Deleting consensus metadata
I20260619 02:56:35.902405 25187 catalog_manager.cc:4841] TS 5ff7a256f9b44f8fb73cdd2a6afd7479 (127.22.230.1:43967): tablet ad663a9ddea745bf930845f891a68cdb (table test-table [id=cf99374142dc480d90fa3164c9c561e5]) successfully deleted
I20260619 02:56:35.903465 25843 ts_tablet_manager.cc:1918] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:56:35.903560 25843 log.cc:1196] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/ad663a9ddea745bf930845f891a68cdb
I20260619 02:56:35.903752 25844 ts_tablet_manager.cc:1918] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 02:56:35.903887 25844 log.cc:1196] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/ad663a9ddea745bf930845f891a68cdb
I20260619 02:56:35.904078 25843 ts_tablet_manager.cc:1939] T ad663a9ddea745bf930845f891a68cdb P eed7cc539e09416986e0121bff049340: Deleting consensus metadata
I20260619 02:56:35.904317 25844 ts_tablet_manager.cc:1939] T ad663a9ddea745bf930845f891a68cdb P 99026503292a48328ae1c2e8c76903bd: Deleting consensus metadata
I20260619 02:56:35.905028 25187 catalog_manager.cc:4841] TS eed7cc539e09416986e0121bff049340 (127.22.230.2:35547): tablet ad663a9ddea745bf930845f891a68cdb (table test-table [id=cf99374142dc480d90fa3164c9c561e5]) successfully deleted
I20260619 02:56:35.905157 25184 catalog_manager.cc:4841] TS 99026503292a48328ae1c2e8c76903bd (127.22.230.3:40751): tablet ad663a9ddea745bf930845f891a68cdb (table test-table [id=cf99374142dc480d90fa3164c9c561e5]) successfully deleted
Jun 19 02:56:35 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:35.915565 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:35.882561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.22.230.1:43967)
0619 02:56:35.888393 (+ 5832us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:35.888403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:35.894797 (+ 6394us) server_negotiation.cc:407] Connection header received
0619 02:56:35.894984 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:35.894989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:35.895066 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:35.895174 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:35.898007 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:35.899203 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:35.901199 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:35.901931 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:35.902028 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:35.909851 (+ 7823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:35.909866 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:35.909882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:35.909918 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:35.912776 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:35.913338 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:35.913349 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:35.913357 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:35.913499 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:35.913794 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:35.913797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:35.913799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:35.914141 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:35.914401 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:35.914657 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 02:56:35.914812 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5646,"thread_start_us":5098,"threads_started":1}
W20260619 02:56:35.916177 25402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40892
W20260619 02:56:35.916743 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25271: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:35.916982 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 25271
I20260619 02:56:35.940342 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25411
Jun 19 02:56:35 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:35.956146 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:35.941094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.22.230.2:35547)
0619 02:56:35.941606 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:35.941612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:35.941630 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:56:35.941689 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:35.941694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:35.941777 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:35.941887 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:35.943062 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:35.944190 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:35.945406 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:35.946032 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:35.946149 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:35.949961 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:35.949977 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:35.949995 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:35.950033 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:35.953537 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:35.954112 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:35.954120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:35.954124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:35.954292 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:35.954712 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:35.954717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:35.954721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:35.955106 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:35.955383 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:35.955560 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 02:56:35.955766 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
W20260619 02:56:35.956857 25541 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126
W20260619 02:56:35.957487 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:35.957631 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 25411
I20260619 02:56:35.982750 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25550
Jun 19 02:56:35 dist-test-slave-2gms krb5kdc[25155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:35.997233 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:35.983406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.22.230.3:40751)
0619 02:56:35.983912 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:35.983916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:35.983930 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:56:35.983980 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:35.983984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:35.984042 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:35.984159 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:35.985387 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:35.986101 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:35.987438 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:35.987827 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:35.987957 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:35.991900 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:35.991913 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:35.991928 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:35.991956 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:35.994425 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:35.995068 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:35.995084 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:35.995087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:35.995208 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:35.995574 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:35.995578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:35.995580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:35.995965 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:35.996358 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:35.996623 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 02:56:35.996840 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":248,"threads_started":1}
W20260619 02:56:35.997900 25680 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60238
W20260619 02:56:35.998436 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:35.998621 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 25550
I20260619 02:56:36.016376 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25168
I20260619 02:56:36.017457 25240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:36.136485 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 25168
2026-06-19T02:56:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8444 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:36 dist-test-slave-2gms krb5kdc[25852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:36 dist-test-slave-2gms krb5kdc[25852](info): set up 2 sockets
Jun 19 02:56:36 dist-test-slave-2gms krb5kdc[25852](info): commencing operation
krb5kdc: starting...
W20260619 02:56:37.211097 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:37 dist-test-slave-2gms krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:37.387758 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:37649
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:37649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:37.694808 25865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:37.695103 25865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:37.695261 25865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:37.701956 25865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:37.702092 25865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:37.702194 25865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:37.702296 25865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:37.702385 25865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:37.711905 25865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:37649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:37.713135 25865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:37.714391 25865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:37.725541 25871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:37.725618 25870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:37.725692 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:37.726213 25865 server_base.cc:1034] running on GCE node
I20260619 02:56:37.727036 25865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:37.728274 25865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:37.729493 25865 hybrid_clock.cc:648] HybridClock initialized: now 1781837797729438 us; error 41 us; skew 500 ppm
I20260619 02:56:37.729813 25865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:37 dist-test-slave-2gms krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837797, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:37.732645 25865 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:37.734028 25865 webserver.cc:460] Webserver started at http://127.22.230.62:33471/ using document root <none> and password file <none>
I20260619 02:56:37.734733 25865 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:37.734870 25865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:37.735153 25865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:37.737892 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1"
format_stamp: "Formatted at 2026-06-19 02:56:37 on dist-test-slave-2gms"
server_key: "dd61fa94098ca26ef87a527a213ec7f0"
server_key_iv: "24676e51121f9cefdad5279c1efec5b1"
server_key_version: "encryptionkey@0"
I20260619 02:56:37.738673 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1"
format_stamp: "Formatted at 2026-06-19 02:56:37 on dist-test-slave-2gms"
server_key: "dd61fa94098ca26ef87a527a213ec7f0"
server_key_iv: "24676e51121f9cefdad5279c1efec5b1"
server_key_version: "encryptionkey@0"
I20260619 02:56:37.743394 25865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:56:37.746407 25880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:37.747630 25865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:56:37.747824 25865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1"
format_stamp: "Formatted at 2026-06-19 02:56:37 on dist-test-slave-2gms"
server_key: "dd61fa94098ca26ef87a527a213ec7f0"
server_key_iv: "24676e51121f9cefdad5279c1efec5b1"
server_key_version: "encryptionkey@0"
I20260619 02:56:37.748008 25865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:37.791003 25865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:37.794735 25865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:37.795004 25865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:37.812836 25865 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:37649
I20260619 02:56:37.812855 25931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:37649 every 8 connection(s)
I20260619 02:56:37.814181 25865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:37.817725 25932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:37.822465 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 25865
I20260619 02:56:37.822718 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:37.823344 23448 external_mini_cluster.cc:1428] Setting key f74bd0be23a68844d25078500b14edda
I20260619 02:56:37.827788 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1: Bootstrap starting.
I20260619 02:56:37.831266 25932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:37.832387 25932 log.cc:824] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:37.834896 25932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1: No bootstrap required, opened a new log
Jun 19 02:56:37 dist-test-slave-2gms krb5kdc[25852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:37.842279 25932 raft_consensus.cc:348] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37649 } }
I20260619 02:56:37.842646 25932 raft_consensus.cc:374] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:37.842777 25932 raft_consensus.cc:729] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e952d0c8f5d54abfbe21f6e5abdb6af1, State: Initialized, Role: FOLLOWER
I20260619 02:56:37.843346 25932 consensus_queue.cc:260] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37649 } }
I20260619 02:56:37.843598 25932 raft_consensus.cc:388] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:37.843811 25932 raft_consensus.cc:482] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:37.844012 25932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:37.844918 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:37.825268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.22.230.62:37649)
0619 02:56:37.826066 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:37.826077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:37.826115 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:56:37.827793 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:37.827814 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:37.828216 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:37.828512 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:37.830267 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:37.831592 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:37.833280 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:37.833912 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:37.834093 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:37.837813 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:37.837819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:37.837832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:37.837869 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:37.840848 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:37.841491 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:37.841502 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:37.841516 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:37.841681 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:37.842045 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:37.842051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:37.842055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:37.842520 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:37.842872 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:37.843467 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 02:56:37.843851 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":369,"threads_started":1}
I20260619 02:56:37.846275 25932 raft_consensus.cc:504] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37649 } }
I20260619 02:56:37.846778 25932 leader_election.cc:302] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e952d0c8f5d54abfbe21f6e5abdb6af1; no voters:
I20260619 02:56:37.847399 25932 leader_election.cc:288] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:37.847509 25937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:37.848235 25937 raft_consensus.cc:686] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [term 1 LEADER]: Becoming Leader. State: Replica: e952d0c8f5d54abfbe21f6e5abdb6af1, State: Running, Role: LEADER
I20260619 02:56:37.848771 25937 consensus_queue.cc:237] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37649 } }
I20260619 02:56:37.849303 25932 sys_catalog.cc:564] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:37.851552 25938 sys_catalog.cc:455] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37649 } } }
I20260619 02:56:37.851980 25938 sys_catalog.cc:458] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:37.853252 25943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:37.853253 25939 sys_catalog.cc:455] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e952d0c8f5d54abfbe21f6e5abdb6af1. Latest consensus state: current_term: 1 leader_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e952d0c8f5d54abfbe21f6e5abdb6af1" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37649 } } }
I20260619 02:56:37.854254 25939 sys_catalog.cc:458] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:37.859503 25943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:37.867395 25943 catalog_manager.cc:1347] Generated new cluster ID: 84967f9d78ba4aaa803fdfab15a3c1e1
I20260619 02:56:37.867508 25943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:37.880640 25943 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:37.881794 25943 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:37.892928 25943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e952d0c8f5d54abfbe21f6e5abdb6af1: Generated new TSK 0
I20260619 02:56:37.893738 25943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:37.945971 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37649
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:38.263046 25958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:38.263293 25958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:38.263381 25958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:38.270174 25958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:38.270269 25958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:38.270371 25958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:38.279175 25958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:38.280220 25958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:38.281334 25958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:38.293485 25963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:38.293620 25966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:38.293648 25964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:38.294467 25958 server_base.cc:1034] running on GCE node
I20260619 02:56:38.295073 25958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:38.295902 25958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:38.297183 25958 hybrid_clock.cc:648] HybridClock initialized: now 1781837798297127 us; error 63 us; skew 500 ppm
I20260619 02:56:38.297514 25958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:38 dist-test-slave-2gms krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837798, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:38.300529 25958 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:38.301785 25958 webserver.cc:460] Webserver started at http://127.22.230.1:41983/ using document root <none> and password file <none>
I20260619 02:56:38.302670 25958 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:38.302778 25958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:38.303049 25958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:38.305735 25958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "72852939832a47da83d3e211ebf4a78f"
format_stamp: "Formatted at 2026-06-19 02:56:38 on dist-test-slave-2gms"
server_key: "2ffbe76095813e9b8cf14c2bc65f670e"
server_key_iv: "ee4eed186831c9d0c3f52e96172c043c"
server_key_version: "encryptionkey@0"
I20260619 02:56:38.306437 25958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "72852939832a47da83d3e211ebf4a78f"
format_stamp: "Formatted at 2026-06-19 02:56:38 on dist-test-slave-2gms"
server_key: "2ffbe76095813e9b8cf14c2bc65f670e"
server_key_iv: "ee4eed186831c9d0c3f52e96172c043c"
server_key_version: "encryptionkey@0"
I20260619 02:56:38.311175 25958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:56:38.314419 25973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:38.315670 25958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:38.315828 25958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "72852939832a47da83d3e211ebf4a78f"
format_stamp: "Formatted at 2026-06-19 02:56:38 on dist-test-slave-2gms"
server_key: "2ffbe76095813e9b8cf14c2bc65f670e"
server_key_iv: "ee4eed186831c9d0c3f52e96172c043c"
server_key_version: "encryptionkey@0"
I20260619 02:56:38.316020 25958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:38.330111 25958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:38.333798 25958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:38.334077 25958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:38.335035 25958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:38.336269 25958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:38.336337 25958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:38.336478 25958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:38.336546 25958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:38.369771 25958 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:44673
I20260619 02:56:38.369855 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:44673 every 8 connection(s)
I20260619 02:56:38.370997 25958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:38.373636 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 25958
I20260619 02:56:38.373837 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:38.374657 23448 external_mini_cluster.cc:1428] Setting key 05d1cd4abfab14b1a6db6601ec754d24
Jun 19 02:56:38 dist-test-slave-2gms krb5kdc[25852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837798, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:38.390196 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:38.373279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:59451 (local address 127.22.230.62:37649)
0619 02:56:38.373698 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:38.373702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:38.374920 (+ 1218us) server_negotiation.cc:407] Connection header received
0619 02:56:38.376861 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:38.376866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:38.376944 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:38.377050 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:38.378439 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.379175 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.380613 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.380993 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.381118 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:38.384891 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:38.384896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:38.384899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:38.384926 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:38.387407 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:38.388073 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:38.388080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:38.388084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:38.388197 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:38.388679 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:38.388686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:38.388690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:38.388939 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:38.389087 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:38.389689 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 02:56:38.389871 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":239,"threads_started":1}
I20260619 02:56:38.391167 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:38.373952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:37649 (local address 127.22.230.1:59451)
0619 02:56:38.374759 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:38.374810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:38.376602 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:38.377283 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:38.377291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:38.377764 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:38.378282 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:38.378297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.379322 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.379333 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:38.380474 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:38.380485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.381196 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.381205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:38.381525 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:38.382198 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:38.382229 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:38.384688 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:38.387537 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:38.387548 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:38.387564 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:38.387948 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:38.388302 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:38.388308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:38.388312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:38.388562 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:38.389065 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:38.389072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:38.389380 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 02:56:38.389621 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 02:56:38.390073 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":301,"threads_started":1}
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:56:38.392697 26086 heartbeater.cc:347] Connected to a master server at 127.22.230.62:37649
I20260619 02:56:38.392992 26086 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:38.393744 26086 heartbeater.cc:511] Master 127.22.230.62:37649 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:56:38.395908 25897 ts_manager.cc:194] Registered new tserver with Master: 72852939832a47da83d3e211ebf4a78f (127.22.230.1:44673)
I20260619 02:56:38.397332 25897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:59451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:38.415462 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37649
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:38.725912 26092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:38.726167 26092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:38.726332 26092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:38.733429 26092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:38.733522 26092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:38.733681 26092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:38.743382 26092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:38.744546 26092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:38.745738 26092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:38.758448 26098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:38.758494 26097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:38.758407 26100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:38.758956 26092 server_base.cc:1034] running on GCE node
I20260619 02:56:38.759626 26092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:38.760555 26092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:38.761754 26092 hybrid_clock.cc:648] HybridClock initialized: now 1781837798761706 us; error 49 us; skew 500 ppm
I20260619 02:56:38.762050 26092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:38 dist-test-slave-2gms krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837798, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:38.764961 26092 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:38.766218 26092 webserver.cc:460] Webserver started at http://127.22.230.2:39743/ using document root <none> and password file <none>
I20260619 02:56:38.766883 26092 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:38.766992 26092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:38.767243 26092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:38.769850 26092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "e1218f293330464ca12e812a27733b7d"
format_stamp: "Formatted at 2026-06-19 02:56:38 on dist-test-slave-2gms"
server_key: "2ddc74bcab08da044c1f9327a13a5b02"
server_key_iv: "1d29791e1c7799c63ce4844637369708"
server_key_version: "encryptionkey@0"
I20260619 02:56:38.770571 26092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "e1218f293330464ca12e812a27733b7d"
format_stamp: "Formatted at 2026-06-19 02:56:38 on dist-test-slave-2gms"
server_key: "2ddc74bcab08da044c1f9327a13a5b02"
server_key_iv: "1d29791e1c7799c63ce4844637369708"
server_key_version: "encryptionkey@0"
I20260619 02:56:38.775368 26092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:56:38.778517 26107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:38.779850 26092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:56:38.780011 26092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "e1218f293330464ca12e812a27733b7d"
format_stamp: "Formatted at 2026-06-19 02:56:38 on dist-test-slave-2gms"
server_key: "2ddc74bcab08da044c1f9327a13a5b02"
server_key_iv: "1d29791e1c7799c63ce4844637369708"
server_key_version: "encryptionkey@0"
I20260619 02:56:38.780205 26092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:38.796715 26092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:38.800454 26092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:38.800720 26092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:38.801574 26092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:38.802820 26092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:38.802915 26092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:38.803066 26092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:38.803192 26092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:38.834558 26092 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:41179
I20260619 02:56:38.834609 26219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:41179 every 8 connection(s)
I20260619 02:56:38.836090 26092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:38.841216 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26092
I20260619 02:56:38.841399 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:38.841925 23448 external_mini_cluster.cc:1428] Setting key 07f65e968122f02e6635b90d8b107128
Jun 19 02:56:38 dist-test-slave-2gms krb5kdc[25852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837798, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:38.857460 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:38.838671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:34279 (local address 127.22.230.62:37649)
0619 02:56:38.838833 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:38.838837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:38.840035 (+ 1198us) server_negotiation.cc:407] Connection header received
0619 02:56:38.842159 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:38.842163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:38.842225 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:38.842309 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:38.843794 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.844492 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.847078 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.847454 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.847543 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:38.851637 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:38.851653 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:38.851657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:38.851685 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:38.854243 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:38.855163 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:38.855171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:38.855177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:38.855317 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:38.855797 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:38.855800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:38.855802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:38.856088 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:38.856207 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:38.857082 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 02:56:38.857252 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:56:38.859071 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:38.839161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:37649 (local address 127.22.230.2:34279)
0619 02:56:38.839881 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:38.839915 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:38.841873 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:38.842612 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:38.842623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:38.843113 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:38.843636 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:38.843651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.844643 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.844650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:38.846931 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:38.846947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:38.847659 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:38.847668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:38.847989 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:38.848697 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:38.848730 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:38.851400 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:38.854371 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:38.854382 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:38.854399 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:38.855030 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:38.855431 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:38.855438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:38.855445 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:38.855677 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:38.856212 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:38.856222 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:38.856660 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 02:56:38.856976 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 02:56:38.857518 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":281,"threads_started":1}
I20260619 02:56:38.860467 26220 heartbeater.cc:347] Connected to a master server at 127.22.230.62:37649
I20260619 02:56:38.860798 26220 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:56:38.861599 26220 heartbeater.cc:511] Master 127.22.230.62:37649 requested a full tablet report, sending...
I20260619 02:56:38.862999 25897 ts_manager.cc:194] Registered new tserver with Master: e1218f293330464ca12e812a27733b7d (127.22.230.2:41179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
I20260619 02:56:38.863937 25897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:34279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:38.883622 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37649
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:39.188740 26225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:39.188992 26225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:39.189087 26225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:39.196015 26225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:39.196116 26225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:39.196220 26225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:39.205268 26225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:37649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:39.206338 26225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:39.207608 26225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:39.219949 26230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:39.219987 26231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:39.220024 26233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:39.220569 26225 server_base.cc:1034] running on GCE node
I20260619 02:56:39.221122 26225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:39.221936 26225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:39.223160 26225 hybrid_clock.cc:648] HybridClock initialized: now 1781837799223128 us; error 44 us; skew 500 ppm
I20260619 02:56:39.223453 26225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:39 dist-test-slave-2gms krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837799, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:39.226318 26225 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:39.227689 26225 webserver.cc:460] Webserver started at http://127.22.230.3:39385/ using document root <none> and password file <none>
I20260619 02:56:39.228303 26225 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:39.228442 26225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:39.228726 26225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:39.231420 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "6e2f090dcb4242f2994e033d3487874e"
format_stamp: "Formatted at 2026-06-19 02:56:39 on dist-test-slave-2gms"
server_key: "e88466ea98c4dd8674c6ae6e5ec10306"
server_key_iv: "d2ddb0d0e74915b9a2b60aafcd17f1e4"
server_key_version: "encryptionkey@0"
I20260619 02:56:39.232197 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "6e2f090dcb4242f2994e033d3487874e"
format_stamp: "Formatted at 2026-06-19 02:56:39 on dist-test-slave-2gms"
server_key: "e88466ea98c4dd8674c6ae6e5ec10306"
server_key_iv: "d2ddb0d0e74915b9a2b60aafcd17f1e4"
server_key_version: "encryptionkey@0"
I20260619 02:56:39.236851 26225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:39.239964 26240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:39.241178 26225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:56:39.241336 26225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "6e2f090dcb4242f2994e033d3487874e"
format_stamp: "Formatted at 2026-06-19 02:56:39 on dist-test-slave-2gms"
server_key: "e88466ea98c4dd8674c6ae6e5ec10306"
server_key_iv: "d2ddb0d0e74915b9a2b60aafcd17f1e4"
server_key_version: "encryptionkey@0"
I20260619 02:56:39.241513 26225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:39.260450 26225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:39.264003 26225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:39.264286 26225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:39.265232 26225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:39.266481 26225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:39.266603 26225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:39.266728 26225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:39.266783 26225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:39.298398 26225 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:44257
I20260619 02:56:39.298473 26352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:44257 every 8 connection(s)
I20260619 02:56:39.299841 26225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:39.308266 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26225
I20260619 02:56:39.308462 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:39.308998 23448 external_mini_cluster.cc:1428] Setting key c2ae4cc0b2eef7ac5eec844474eb292c
Jun 19 02:56:39 dist-test-slave-2gms krb5kdc[25852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837799, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:39.318596 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:39.302176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:54353 (local address 127.22.230.62:37649)
0619 02:56:39.302330 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:39.302335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:39.303512 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 02:56:39.305113 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:39.305117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:39.305180 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:39.305271 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:39.306628 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:39.307361 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:39.309353 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:39.309900 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:39.309987 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:39.313313 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:39.313318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:39.313321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:39.313351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:39.315879 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:39.316500 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:39.316506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:39.316509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:39.316641 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:39.317017 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:39.317024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:39.317027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:39.317268 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:39.317429 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:39.318043 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 02:56:39.318247 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:56:39.319589 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:39.302621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:37649 (local address 127.22.230.3:54353)
0619 02:56:39.303346 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:39.303389 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:39.304882 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:39.305458 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:39.305464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:39.305899 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:39.306423 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:39.306438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:39.307477 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:39.307481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:39.309223 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:39.309237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:39.310110 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:39.310116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:39.310385 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:39.311065 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:39.311092 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:39.313121 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:39.316000 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:39.316008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:39.316021 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:39.316382 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:39.316742 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:39.316747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:39.316750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:39.316913 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:39.317402 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:39.317409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:39.317707 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 02:56:39.317973 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 02:56:39.318416 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":289,"threads_started":1}
I20260619 02:56:39.320786 26353 heartbeater.cc:347] Connected to a master server at 127.22.230.62:37649
I20260619 02:56:39.321054 26353 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:39.321725 26353 heartbeater.cc:511] Master 127.22.230.62:37649 requested a full tablet report, sending...
I20260619 02:56:39.323221 25897 ts_manager.cc:194] Registered new tserver with Master: 6e2f090dcb4242f2994e033d3487874e (127.22.230.3:44257)
I20260619 02:56:39.323951 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:56:39.324038 25897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:54353
W20260619 02:56:39.352998 26088 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:56:39.350522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.22.230.62:37649)
0619 02:56:39.350778 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:39.350788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:39.351207 (+ 419us) server_negotiation.cc:407] Connection header received
0619 02:56:39.351404 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:39.351410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:39.351494 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:39.351609 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:39.352728 (+ 1119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48808: BlockingRecv error: recv got EOF from 127.0.0.1:48808 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 02:56:39.355991 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25958
W20260619 02:56:39.358613 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25958: Not authorized: Client connection negotiation failed: client connection to 127.22.230.1:44673: server requires authentication, but client does not have Kerberos credentials available
I20260619 02:56:39.358795 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 25958
W20260619 02:56:39.358815 26364 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:56:39.356901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.22.230.1:44673)
0619 02:56:39.357360 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:39.357368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:39.357426 (+ 58us) server_negotiation.cc:407] Connection header received
0619 02:56:39.357477 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:39.357483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:39.357682 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:39.357799 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:39.358444 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56276: BlockingRecv error: recv got EOF from 127.0.0.1:56276 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":235,"threads_started":1}
I20260619 02:56:39.374483 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26092
W20260619 02:56:39.376742 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26092: Not authorized: Client connection negotiation failed: client connection to 127.22.230.2:41179: server requires authentication, but client does not have Kerberos credentials available
I20260619 02:56:39.376904 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26092
I20260619 02:56:39.391692 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26225
W20260619 02:56:39.394109 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26225: Not authorized: Client connection negotiation failed: client connection to 127.22.230.3:44257: server requires authentication, but client does not have Kerberos credentials available
I20260619 02:56:39.394235 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26225
I20260619 02:56:39.408771 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 25865
I20260619 02:56:39.409765 25927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:39.509657 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 25865
2026-06-19T02:56:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3372 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:39 dist-test-slave-2gms krb5kdc[26370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:39 dist-test-slave-2gms krb5kdc[26370](info): set up 2 sockets
Jun 19 02:56:39 dist-test-slave-2gms krb5kdc[26370](info): commencing operation
krb5kdc: starting...
W20260619 02:56:40.628566 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 02:56:40.941780 23448 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.313s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 02:56:41.194636 23448 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.253s user 0.000s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 02:56:41.350317 23448 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.155s user 0.003s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/test-user.keytab.
W20260619 02:56:41.474732 23448 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.124s user 0.001s sys 0.004s
Jun 19 02:56:41 dist-test-slave-2gms krb5kdc[26370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:41.557663 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:40953
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:40953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:41.862293 26383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:41.862622 26383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:41.862762 26383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:41.869478 26383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:41.869568 26383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:41.869612 26383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:41.869666 26383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:41.869704 26383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:41.878736 26383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:40953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:41.880103 26383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:41.881265 26383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:41.892374 26388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:41.892351 26389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:41.892468 26391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:41.893224 26383 server_base.cc:1034] running on GCE node
I20260619 02:56:41.893971 26383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:41.895323 26383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:41.896538 26383 hybrid_clock.cc:648] HybridClock initialized: now 1781837801896509 us; error 42 us; skew 500 ppm
I20260619 02:56:41.896868 26383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:41 dist-test-slave-2gms krb5kdc[26370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837801, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:41.904304 26383 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:41.905722 26383 webserver.cc:460] Webserver started at http://127.22.230.62:35855/ using document root <none> and password file <none>
I20260619 02:56:41.906368 26383 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:41.906505 26383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:41.906829 26383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:41.909581 26383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "c3ef5ca66f1743a8825848f8a7b5797b"
format_stamp: "Formatted at 2026-06-19 02:56:41 on dist-test-slave-2gms"
server_key: "127fd961aece9c709cf93cce4e13f53f"
server_key_iv: "9a1ee779c2054d291313a0ab42dbcc8e"
server_key_version: "encryptionkey@0"
I20260619 02:56:41.910322 26383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "c3ef5ca66f1743a8825848f8a7b5797b"
format_stamp: "Formatted at 2026-06-19 02:56:41 on dist-test-slave-2gms"
server_key: "127fd961aece9c709cf93cce4e13f53f"
server_key_iv: "9a1ee779c2054d291313a0ab42dbcc8e"
server_key_version: "encryptionkey@0"
I20260619 02:56:41.915107 26383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:56:41.918176 26398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:41.919397 26383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:56:41.919570 26383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "c3ef5ca66f1743a8825848f8a7b5797b"
format_stamp: "Formatted at 2026-06-19 02:56:41 on dist-test-slave-2gms"
server_key: "127fd961aece9c709cf93cce4e13f53f"
server_key_iv: "9a1ee779c2054d291313a0ab42dbcc8e"
server_key_version: "encryptionkey@0"
I20260619 02:56:41.919808 26383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:41.931964 26383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:41.935648 26383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:41.935917 26383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:41.953717 26383 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:40953
I20260619 02:56:41.953761 26449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:40953 every 8 connection(s)
I20260619 02:56:41.955003 26383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:41.958391 26450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:41.964600 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26383
I20260619 02:56:41.964811 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:41.965610 23448 external_mini_cluster.cc:1428] Setting key 3855f34b84e4b65ab6d316e46439df15
I20260619 02:56:41.969188 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b: Bootstrap starting.
I20260619 02:56:41.972767 26450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:41.973778 26450 log.cc:824] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:41.976734 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b: No bootstrap required, opened a new log
Jun 19 02:56:41 dist-test-slave-2gms krb5kdc[26370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:41.985093 26450 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40953 } }
I20260619 02:56:41.985446 26450 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:41.985574 26450 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ef5ca66f1743a8825848f8a7b5797b, State: Initialized, Role: FOLLOWER
I20260619 02:56:41.986248 26450 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40953 } }
I20260619 02:56:41.986507 26450 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:41.986718 26450 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:41.986912 26450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:41.987685 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:41.967440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.22.230.62:40953)
0619 02:56:41.968174 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:41.968186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:41.968224 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:56:41.969906 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:41.969927 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:41.970312 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:41.970722 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:41.972604 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:41.973966 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:41.975617 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:41.976257 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:41.976391 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:41.980724 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:41.980732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:41.980745 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:41.980776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:41.983804 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:41.984479 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:41.984486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:41.984492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:41.984643 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:41.985004 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:41.985013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:41.985015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:41.985534 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:41.985833 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:41.986210 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 02:56:41.986562 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":337,"threads_started":1}
I20260619 02:56:41.989243 26450 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40953 } }
I20260619 02:56:41.989713 26450 leader_election.cc:302] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3ef5ca66f1743a8825848f8a7b5797b; no voters:
I20260619 02:56:41.990302 26450 leader_election.cc:288] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:41.990407 26455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:41.991411 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:56:41.991520 26455 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [term 1 LEADER]: Becoming Leader. State: Replica: c3ef5ca66f1743a8825848f8a7b5797b, State: Running, Role: LEADER
I20260619 02:56:41.991981 26455 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40953 } }
I20260619 02:56:41.992344 26450 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:41.994776 26456 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40953 } } }
I20260619 02:56:41.995169 26456 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:41.995697 26457 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3ef5ca66f1743a8825848f8a7b5797b. Latest consensus state: current_term: 1 leader_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef5ca66f1743a8825848f8a7b5797b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40953 } } }
I20260619 02:56:41.995966 26457 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:42.001389 26460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:42.010440 26460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:42.020246 26460 catalog_manager.cc:1347] Generated new cluster ID: ae7ffc53331d4a17b18d24448d4ab025
I20260619 02:56:42.020385 26460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:42.035631 26460 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:42.037118 26460 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:42.045285 26460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3ef5ca66f1743a8825848f8a7b5797b: Generated new TSK 0
I20260619 02:56:42.046208 26460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:56:42.046715 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:42.036467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.22.230.62:40953)
0619 02:56:42.036616 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:42.036622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:42.039104 (+ 2482us) server_negotiation.cc:407] Connection header received
0619 02:56:42.039244 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:42.039250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:42.039329 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:42.039436 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:42.040594 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:42.041714 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:42.043436 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:42.043875 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:42.044021 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:42.045172 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:42.045179 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:42.045183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:42.045867 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:42.045924 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:42.045945 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:42.046015 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:42.046216 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 02:56:42.046356 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260619 02:56:42.062202 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26383
I20260619 02:56:42.063403 26445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:42.151321 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26383
2026-06-19T02:56:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2634 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:42 dist-test-slave-2gms krb5kdc[26484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:42 dist-test-slave-2gms krb5kdc[26484](info): set up 2 sockets
Jun 19 02:56:42 dist-test-slave-2gms krb5kdc[26484](info): commencing operation
krb5kdc: starting...
W20260619 02:56:43.218735 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:43 dist-test-slave-2gms krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:43.395740 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35811
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:43.696605 26497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:43.696907 26497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:43.697077 26497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:43.704090 26497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:43.704219 26497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:43.704322 26497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:43.704422 26497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:43.704506 26497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:43.713742 26497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:35811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:43.714957 26497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:43.716192 26497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:43.727530 26505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:43.727612 26502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:43.727586 26503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:43.728389 26497 server_base.cc:1034] running on GCE node
I20260619 02:56:43.729070 26497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:43.730304 26497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:43.731536 26497 hybrid_clock.cc:648] HybridClock initialized: now 1781837803731495 us; error 49 us; skew 500 ppm
I20260619 02:56:43.731863 26497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:43 dist-test-slave-2gms krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837803, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:43.734740 26497 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:43.735994 26497 webserver.cc:460] Webserver started at http://127.22.230.62:34195/ using document root <none> and password file <none>
I20260619 02:56:43.736599 26497 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:43.736738 26497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:43.737016 26497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:43.739778 26497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "4e9b586da31a4c37a83e4191204ede5f"
format_stamp: "Formatted at 2026-06-19 02:56:43 on dist-test-slave-2gms"
server_key: "4d6bf74a8bebb42689f145e9f218a0ec"
server_key_iv: "e7236f08d753a2f8f266b8fbb91f1c57"
server_key_version: "encryptionkey@0"
I20260619 02:56:43.740484 26497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "4e9b586da31a4c37a83e4191204ede5f"
format_stamp: "Formatted at 2026-06-19 02:56:43 on dist-test-slave-2gms"
server_key: "4d6bf74a8bebb42689f145e9f218a0ec"
server_key_iv: "e7236f08d753a2f8f266b8fbb91f1c57"
server_key_version: "encryptionkey@0"
I20260619 02:56:43.745020 26497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:43.748137 26512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:43.749284 26497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:56:43.749460 26497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "4e9b586da31a4c37a83e4191204ede5f"
format_stamp: "Formatted at 2026-06-19 02:56:43 on dist-test-slave-2gms"
server_key: "4d6bf74a8bebb42689f145e9f218a0ec"
server_key_iv: "e7236f08d753a2f8f266b8fbb91f1c57"
server_key_version: "encryptionkey@0"
I20260619 02:56:43.749635 26497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:43.779256 26497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:43.782876 26497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:43.783154 26497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:43.800482 26497 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:35811
I20260619 02:56:43.800566 26563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:35811 every 8 connection(s)
I20260619 02:56:43.801666 26497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:43.804903 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:43.811030 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26497
I20260619 02:56:43.811235 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:43.811990 23448 external_mini_cluster.cc:1428] Setting key 6741dd60a1c19e0ca3db6fc3d8328ac6
I20260619 02:56:43.815421 26564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f: Bootstrap starting.
I20260619 02:56:43.819247 26564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:43.820262 26564 log.cc:824] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:43.823271 26564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f: No bootstrap required, opened a new log
Jun 19 02:56:43 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:43.833410 26564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9b586da31a4c37a83e4191204ede5f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35811 } }
I20260619 02:56:43.833850 26564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:43.833948 26564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b586da31a4c37a83e4191204ede5f, State: Initialized, Role: FOLLOWER
I20260619 02:56:43.834594 26564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9b586da31a4c37a83e4191204ede5f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35811 } }
I20260619 02:56:43.834867 26564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:43.834997 26564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:43.835031 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:43.813902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.22.230.62:35811)
0619 02:56:43.814661 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:43.814672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:43.814710 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:56:43.816333 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:43.816355 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:43.816705 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:43.816993 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:43.818848 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:43.820075 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:43.821625 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:43.822236 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:43.822364 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:43.826943 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:43.826972 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:43.827007 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:43.827053 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:43.830843 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:43.831499 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:43.831512 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:43.831520 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:43.831718 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:43.832043 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:43.832048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:43.832051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:43.832699 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:43.833000 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:43.833259 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:56:43.833684 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":348,"threads_started":1}
I20260619 02:56:43.835179 26564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:43.837240 26564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9b586da31a4c37a83e4191204ede5f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35811 } }
I20260619 02:56:43.837790 26564 leader_election.cc:302] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e9b586da31a4c37a83e4191204ede5f; no voters:
I20260619 02:56:43.838341 26564 leader_election.cc:288] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:43.838431 26569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:43.839615 26569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [term 1 LEADER]: Becoming Leader. State: Replica: 4e9b586da31a4c37a83e4191204ede5f, State: Running, Role: LEADER
I20260619 02:56:43.840180 26569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9b586da31a4c37a83e4191204ede5f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35811 } }
I20260619 02:56:43.840421 26564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:43.842592 26570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e9b586da31a4c37a83e4191204ede5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9b586da31a4c37a83e4191204ede5f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35811 } } }
I20260619 02:56:43.842923 26570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:43.842798 26571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e9b586da31a4c37a83e4191204ede5f. Latest consensus state: current_term: 1 leader_uuid: "4e9b586da31a4c37a83e4191204ede5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9b586da31a4c37a83e4191204ede5f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35811 } } }
I20260619 02:56:43.843063 26571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:43.843719 26574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:43.848707 26574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:43.858321 26574 catalog_manager.cc:1347] Generated new cluster ID: 68afdfe68efc47d0b58064d908e4eaf5
I20260619 02:56:43.858415 26574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:43.900012 26574 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:43.901271 26574 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:43.910115 26574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e9b586da31a4c37a83e4191204ede5f: Generated new TSK 0
I20260619 02:56:43.910887 26574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:43.955978 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35811
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:44.270614 26590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:44.270874 26590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:44.271039 26590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:44.277978 26590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:44.278123 26590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:44.278309 26590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:44.287827 26590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:44.289094 26590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:44.290382 26590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:44.301841 26595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:44.302839 26596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:44.303352 26598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:44.304368 26590 server_base.cc:1034] running on GCE node
I20260619 02:56:44.304929 26590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:44.305831 26590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:44.307097 26590 hybrid_clock.cc:648] HybridClock initialized: now 1781837804307050 us; error 57 us; skew 500 ppm
I20260619 02:56:44.307417 26590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:44 dist-test-slave-2gms krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:44.310392 26590 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:44.311736 26590 webserver.cc:460] Webserver started at http://127.22.230.1:37277/ using document root <none> and password file <none>
I20260619 02:56:44.312449 26590 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:44.312587 26590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:44.312867 26590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:44.315711 26590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "7c2a8ec602374b51b32f5b51d0eadb2d"
format_stamp: "Formatted at 2026-06-19 02:56:44 on dist-test-slave-2gms"
server_key: "0fbb1316bf43f43ab8d508c1eda87b52"
server_key_iv: "97bc1c45b490b7b0da90984896737448"
server_key_version: "encryptionkey@0"
I20260619 02:56:44.316455 26590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "7c2a8ec602374b51b32f5b51d0eadb2d"
format_stamp: "Formatted at 2026-06-19 02:56:44 on dist-test-slave-2gms"
server_key: "0fbb1316bf43f43ab8d508c1eda87b52"
server_key_iv: "97bc1c45b490b7b0da90984896737448"
server_key_version: "encryptionkey@0"
I20260619 02:56:44.321301 26590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:56:44.324784 26605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:44.325975 26590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:56:44.326143 26590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "7c2a8ec602374b51b32f5b51d0eadb2d"
format_stamp: "Formatted at 2026-06-19 02:56:44 on dist-test-slave-2gms"
server_key: "0fbb1316bf43f43ab8d508c1eda87b52"
server_key_iv: "97bc1c45b490b7b0da90984896737448"
server_key_version: "encryptionkey@0"
I20260619 02:56:44.326330 26590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:44.337733 26590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:44.341651 26590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:44.341952 26590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:44.342921 26590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:44.344121 26590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:44.344226 26590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:44.344343 26590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:44.344429 26590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:44.376828 26590 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:33913
I20260619 02:56:44.376875 26717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:33913 every 8 connection(s)
I20260619 02:56:44.378003 26590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:44.383288 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26590
I20260619 02:56:44.383493 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:44.384351 23448 external_mini_cluster.cc:1428] Setting key 2591393c9569de1092ff22ebc7825178
Jun 19 02:56:44 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:44.398289 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:44.380361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:38709 (local address 127.22.230.62:35811)
0619 02:56:44.380807 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:44.380812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:44.381648 (+ 836us) server_negotiation.cc:407] Connection header received
0619 02:56:44.383434 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:44.383439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:44.383549 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:44.383675 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:44.385468 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.386267 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.388755 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.389131 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.389217 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:44.393025 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:44.393030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:44.393033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:44.393061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:44.395556 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:44.396202 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:44.396212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:44.396215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:44.396315 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:44.396735 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:44.396740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:44.396743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:44.397029 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:44.397169 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:44.397754 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 02:56:44.397948 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260619 02:56:44.399204 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:44.380853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35811 (local address 127.22.230.1:38709)
0619 02:56:44.381510 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:44.381546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:44.383190 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:44.383938 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:44.383951 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:44.384674 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:44.385307 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:44.385332 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.386398 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.386404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:44.388628 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:44.388645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.389329 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.389338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:44.389651 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:44.390322 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:44.390352 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:44.392820 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:44.395688 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:44.395695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:44.395709 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:44.396084 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:44.396441 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:44.396446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:44.396450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:44.396624 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:44.397144 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:44.397151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:44.397431 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 02:56:44.397666 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 02:56:44.398134 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":246,"threads_started":1}
I20260619 02:56:44.400712 26718 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35811
I20260619 02:56:44.401001 26718 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:44.401674 26718 heartbeater.cc:511] Master 127.22.230.62:35811 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:56:44.403643 26529 ts_manager.cc:194] Registered new tserver with Master: 7c2a8ec602374b51b32f5b51d0eadb2d (127.22.230.1:33913)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:56:44.405145 26529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:38709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:44.427182 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35811
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:44.734923 26724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:44.735158 26724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:44.735316 26724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:44.742254 26724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:44.742367 26724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:44.742543 26724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:44.751566 26724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:44.752797 26724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:44.754063 26724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:44.766559 26729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:44.767329 26730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:44.767386 26732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:44.767693 26724 server_base.cc:1034] running on GCE node
I20260619 02:56:44.768326 26724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:44.769181 26724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:44.770372 26724 hybrid_clock.cc:648] HybridClock initialized: now 1781837804770337 us; error 27 us; skew 500 ppm
I20260619 02:56:44.770680 26724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:44 dist-test-slave-2gms krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:44.773589 26724 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:44.775003 26724 webserver.cc:460] Webserver started at http://127.22.230.2:42661/ using document root <none> and password file <none>
I20260619 02:56:44.775635 26724 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:44.775765 26724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:44.776052 26724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:44.778868 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "eba75576bb5147eabeb8debd22281f58"
format_stamp: "Formatted at 2026-06-19 02:56:44 on dist-test-slave-2gms"
server_key: "156cadab89b8b0d9cd5b319bc35b0c90"
server_key_iv: "9ce7acdbcb5e45bee3c5775ae62fb911"
server_key_version: "encryptionkey@0"
I20260619 02:56:44.779593 26724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "eba75576bb5147eabeb8debd22281f58"
format_stamp: "Formatted at 2026-06-19 02:56:44 on dist-test-slave-2gms"
server_key: "156cadab89b8b0d9cd5b319bc35b0c90"
server_key_iv: "9ce7acdbcb5e45bee3c5775ae62fb911"
server_key_version: "encryptionkey@0"
I20260619 02:56:44.784337 26724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:44.787668 26739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:44.788892 26724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:56:44.789048 26724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "eba75576bb5147eabeb8debd22281f58"
format_stamp: "Formatted at 2026-06-19 02:56:44 on dist-test-slave-2gms"
server_key: "156cadab89b8b0d9cd5b319bc35b0c90"
server_key_iv: "9ce7acdbcb5e45bee3c5775ae62fb911"
server_key_version: "encryptionkey@0"
I20260619 02:56:44.789228 26724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:44.808806 26724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:44.812801 26724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:44.813069 26724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:44.814034 26724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:44.815346 26724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:44.815443 26724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:44.815590 26724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:44.815665 26724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:44.847833 26724 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:44279
I20260619 02:56:44.847896 26851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:44279 every 8 connection(s)
I20260619 02:56:44.849007 26724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:44.852037 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26724
I20260619 02:56:44.852193 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:44.852671 23448 external_mini_cluster.cc:1428] Setting key 3f468781a3929af3e7711bb1e97126ba
Jun 19 02:56:44 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837804, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:44.869691 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:44.851441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:45977 (local address 127.22.230.62:35811)
0619 02:56:44.851610 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:44.851615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:44.852804 (+ 1189us) server_negotiation.cc:407] Connection header received
0619 02:56:44.854896 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:44.854901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:44.854985 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:44.855090 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:44.856416 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.857522 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.859002 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.859647 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.859798 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:44.863079 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:44.863087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:44.863094 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:44.863134 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:44.866518 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:44.867225 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:44.867234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:44.867240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:44.867392 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:44.867775 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:44.867783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:44.867789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:44.868149 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:44.868319 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:44.869033 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 02:56:44.869277 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:56:44.870190 26854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:44.851874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35811 (local address 127.22.230.2:45977)
0619 02:56:44.852661 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:44.852702 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:44.854648 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:44.855268 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:44.855276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:44.855756 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:44.856261 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:44.856275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.857693 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.857697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:44.858882 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:44.858892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:44.859852 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:44.859856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:44.860035 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:44.860705 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:44.860728 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:44.862879 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:44.866665 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:44.866671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:44.866681 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:44.867096 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:44.867503 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:44.867505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:44.867507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:44.867660 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:44.868273 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:44.868278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:44.868689 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 02:56:44.868878 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 02:56:44.869191 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":354,"threads_started":1}
I20260619 02:56:44.871621 26852 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35811
I20260619 02:56:44.871958 26852 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:56:44.872812 26852 heartbeater.cc:511] Master 127.22.230.62:35811 requested a full tablet report, sending...
I20260619 02:56:44.874094 26529 ts_manager.cc:194] Registered new tserver with Master: eba75576bb5147eabeb8debd22281f58 (127.22.230.2:44279)
I20260619 02:56:44.874933 26529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:45977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:44.897197 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35811
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:45.201671 26857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:45.201970 26857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:45.202139 26857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:45.208935 26857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:45.209069 26857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:45.209245 26857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:45.217952 26857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35811
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:45.219127 26857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:45.220357 26857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:45.232820 26865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:45.232792 26863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:45.232937 26862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:45.233821 26857 server_base.cc:1034] running on GCE node
I20260619 02:56:45.234359 26857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:45.235239 26857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:45.236416 26857 hybrid_clock.cc:648] HybridClock initialized: now 1781837805236388 us; error 27 us; skew 500 ppm
I20260619 02:56:45.236699 26857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:45.239602 26857 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:45.240998 26857 webserver.cc:460] Webserver started at http://127.22.230.3:42521/ using document root <none> and password file <none>
I20260619 02:56:45.241742 26857 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:45.241883 26857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:45.242188 26857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:45.244863 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "2ffe0f81f620419983a30d55f7763678"
format_stamp: "Formatted at 2026-06-19 02:56:45 on dist-test-slave-2gms"
server_key: "81d34e125032186fa4b49ad64c0d35ae"
server_key_iv: "709c838a1a0d21f687a8845ceac16666"
server_key_version: "encryptionkey@0"
I20260619 02:56:45.245569 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "2ffe0f81f620419983a30d55f7763678"
format_stamp: "Formatted at 2026-06-19 02:56:45 on dist-test-slave-2gms"
server_key: "81d34e125032186fa4b49ad64c0d35ae"
server_key_iv: "709c838a1a0d21f687a8845ceac16666"
server_key_version: "encryptionkey@0"
I20260619 02:56:45.250346 26857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 02:56:45.253574 26872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:45.254827 26857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:56:45.254987 26857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "2ffe0f81f620419983a30d55f7763678"
format_stamp: "Formatted at 2026-06-19 02:56:45 on dist-test-slave-2gms"
server_key: "81d34e125032186fa4b49ad64c0d35ae"
server_key_iv: "709c838a1a0d21f687a8845ceac16666"
server_key_version: "encryptionkey@0"
I20260619 02:56:45.255170 26857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:45.285667 26857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:45.289490 26857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:45.289757 26857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:45.290655 26857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:45.291853 26857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:45.291985 26857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:45.292110 26857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:45.292199 26857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:45.323247 26857 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40763
I20260619 02:56:45.323285 26984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40763 every 8 connection(s)
I20260619 02:56:45.324571 26857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:45.332949 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 26857
I20260619 02:56:45.333127 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:45.333578 23448 external_mini_cluster.cc:1428] Setting key abf964387a1832458e9eb0fc66271f84
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:45.343384 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:45.327115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:44725 (local address 127.22.230.62:35811)
0619 02:56:45.327285 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:45.327290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:45.328355 (+ 1065us) server_negotiation.cc:407] Connection header received
0619 02:56:45.330141 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:45.330146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:45.330233 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:45.330356 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:45.331796 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.332510 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.334171 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.334787 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.334897 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:45.338160 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:45.338165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:45.338170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:45.338210 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:45.340619 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.341247 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.341253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.341256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.341370 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.341762 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.341770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.341773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.342044 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:45.342188 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:45.342846 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 02:56:45.343044 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:56:45.344201 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:45.327546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35811 (local address 127.22.230.3:44725)
0619 02:56:45.328209 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:45.328244 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:45.329884 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:45.330619 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:45.330635 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:45.331115 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:45.331644 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:45.331660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.332656 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.332664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:45.334031 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:45.334043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.334982 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.334990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:45.335212 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:45.335947 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:45.335995 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:45.337964 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:45.340744 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:45.340751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:45.340765 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:45.341134 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:45.341469 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:45.341474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:45.341479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:45.341644 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:45.342161 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:45.342168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:45.342462 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 02:56:45.342764 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 02:56:45.343218 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":260,"threads_started":1}
I20260619 02:56:45.345376 26985 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35811
I20260619 02:56:45.345597 26985 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:45.346176 26985 heartbeater.cc:511] Master 127.22.230.62:35811 requested a full tablet report, sending...
I20260619 02:56:45.347544 26529 ts_manager.cc:194] Registered new tserver with Master: 2ffe0f81f620419983a30d55f7763678 (127.22.230.3:40763)
I20260619 02:56:45.348508 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:56:45.348537 26529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:44725
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:45.394560 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:45.382429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.22.230.62:35811)
0619 02:56:45.382669 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:45.382674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:45.382985 (+ 311us) server_negotiation.cc:407] Connection header received
0619 02:56:45.383123 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:45.383126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:45.383185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:45.383273 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:45.384466 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.385167 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.386267 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.386722 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.386813 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:45.390178 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:45.390182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:45.390186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:45.390213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:45.392521 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.393021 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.393025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.393027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.393132 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.393495 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.393498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.393500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.393816 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:45.393979 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:45.394203 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:56:45.394342 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 02:56:45.395186 26529 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33020
I20260619 02:56:45.397166 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26590
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:45.408578 26718 heartbeater.cc:503] Master 127.22.230.62:35811 was elected leader, sending a full tablet report...
I20260619 02:56:45.411819 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:45.397933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.22.230.1:33913)
0619 02:56:45.398462 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:45.398472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:45.398496 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:56:45.398613 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:45.398622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:45.398820 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:45.398964 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:45.399964 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.400927 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.402100 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.402686 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.402829 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:45.406329 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:45.406339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:45.406350 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:45.406382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:45.409191 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.409809 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.409817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.409821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.409955 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.410302 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.410308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.410312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.410687 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:45.410960 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:45.411245 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 02:56:45.411487 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":253,"threads_started":1}
W20260619 02:56:45.412890 26713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49958
W20260619 02:56:45.413897 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26590: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:45.414105 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26590
I20260619 02:56:45.429312 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26724
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:45.444056 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:45.430093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.22.230.2:44279)
0619 02:56:45.430662 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:45.430668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:45.430691 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:56:45.430747 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:45.430752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:45.430934 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:45.431065 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:45.432119 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.433033 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.434173 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.434742 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.434833 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:45.438530 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:45.438540 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:45.438551 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:45.438634 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:45.441375 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.441964 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.441974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.441977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.442091 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.442474 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.442480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.442483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.442895 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:45.443174 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:45.443454 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 02:56:45.443732 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":280,"threads_started":1}
W20260619 02:56:45.445168 26847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40094
W20260619 02:56:45.446112 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:45.446249 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26724
I20260619 02:56:45.461678 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26857
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[26484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837805, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:45.476944 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:45.462486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.22.230.3:40763)
0619 02:56:45.463081 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:45.463087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:45.463114 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:56:45.463166 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:45.463172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:45.463356 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:45.463517 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:45.464773 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.465771 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.467104 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:45.467628 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:45.467768 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:45.471803 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:45.471813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:45.471825 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:45.471852 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:45.474279 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.474952 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.474964 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.474966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.475072 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:45.475426 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:45.475430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:45.475433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:45.475739 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:45.476026 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:45.476359 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 02:56:45.476622 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":326,"threads_started":1}
W20260619 02:56:45.478103 26980 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34958
W20260619 02:56:45.479142 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26857: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 02:56:45.479274 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26857
I20260619 02:56:45.494202 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 26497
I20260619 02:56:45.495200 26559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:45.594862 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 26497
2026-06-19T02:56:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3448 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[27002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[27002](info): set up 2 sockets
Jun 19 02:56:45 dist-test-slave-2gms krb5kdc[27002](info): commencing operation
krb5kdc: starting...
W20260619 02:56:46.694988 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:46 dist-test-slave-2gms krb5kdc[27002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:46.890491 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35583
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:35583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:47.194547 27015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:47.194887 27015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:47.195041 27015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:47.201992 27015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:47.202136 27015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:47.202250 27015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:47.202343 27015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:47.202426 27015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:47.211881 27015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:35583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:47.213135 27015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:47.214390 27015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:47.225749 27020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:47.225852 27023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:47.225909 27021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:47.226851 27015 server_base.cc:1034] running on GCE node
I20260619 02:56:47.227600 27015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:47.228902 27015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:47.230091 27015 hybrid_clock.cc:648] HybridClock initialized: now 1781837807230056 us; error 30 us; skew 500 ppm
I20260619 02:56:47.230427 27015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:47 dist-test-slave-2gms krb5kdc[27002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:47.236331 27015 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:47.237743 27015 webserver.cc:460] Webserver started at http://127.22.230.62:39699/ using document root <none> and password file <none>
I20260619 02:56:47.238401 27015 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:47.238525 27015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:47.238862 27015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:47.241623 27015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "990a34a24943459b8220854089c2b29e"
format_stamp: "Formatted at 2026-06-19 02:56:47 on dist-test-slave-2gms"
server_key: "7db359ed9793469f1422f9d5990953eb"
server_key_iv: "42994b1af3b8dbd46b1b14a01b463492"
server_key_version: "encryptionkey@0"
I20260619 02:56:47.242362 27015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "990a34a24943459b8220854089c2b29e"
format_stamp: "Formatted at 2026-06-19 02:56:47 on dist-test-slave-2gms"
server_key: "7db359ed9793469f1422f9d5990953eb"
server_key_iv: "42994b1af3b8dbd46b1b14a01b463492"
server_key_version: "encryptionkey@0"
I20260619 02:56:47.247156 27015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:56:47.250283 27030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:47.251830 27015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:47.251998 27015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "990a34a24943459b8220854089c2b29e"
format_stamp: "Formatted at 2026-06-19 02:56:47 on dist-test-slave-2gms"
server_key: "7db359ed9793469f1422f9d5990953eb"
server_key_iv: "42994b1af3b8dbd46b1b14a01b463492"
server_key_version: "encryptionkey@0"
I20260619 02:56:47.252182 27015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:47.264788 27015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:47.268427 27015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:47.268718 27015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:47.285382 27015 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:35583
I20260619 02:56:47.285403 27081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:35583 every 8 connection(s)
I20260619 02:56:47.286778 27015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:47.288794 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27015
I20260619 02:56:47.289016 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:47.289790 23448 external_mini_cluster.cc:1428] Setting key 579973c7bdb96cb53e08d3ffb32379c1
I20260619 02:56:47.291208 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:56:47 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:47.304617 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e: Bootstrap starting.
I20260619 02:56:47.307570 27082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:47.308523 27082 log.cc:824] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:47.311324 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e: No bootstrap required, opened a new log
I20260619 02:56:47.311447 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:47.291749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55260 (local address 127.22.230.62:35583)
0619 02:56:47.292472 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:47.292483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:47.292521 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:56:47.293991 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:47.294011 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:47.294375 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:47.294760 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:47.296455 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:47.297771 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:47.299512 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:47.300133 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:47.300247 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:47.304624 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:47.304631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:47.304645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:47.304675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:47.307595 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:47.308220 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:47.308234 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:47.308239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:47.308400 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:47.308779 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:47.308784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:47.308787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:47.309265 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:47.309580 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:47.310013 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:56:47.310402 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":354,"threads_started":1}
I20260619 02:56:47.320024 27082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990a34a24943459b8220854089c2b29e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35583 } }
I20260619 02:56:47.320369 27082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:47.320470 27082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990a34a24943459b8220854089c2b29e, State: Initialized, Role: FOLLOWER
I20260619 02:56:47.321096 27082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990a34a24943459b8220854089c2b29e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35583 } }
I20260619 02:56:47.321350 27082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:47.321480 27082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:47.321612 27082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:47.323649 27082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990a34a24943459b8220854089c2b29e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35583 } }
I20260619 02:56:47.324074 27082 leader_election.cc:302] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 990a34a24943459b8220854089c2b29e; no voters:
I20260619 02:56:47.324564 27082 leader_election.cc:288] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:47.324684 27087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:47.326387 27082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:47.328485 27087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [term 1 LEADER]: Becoming Leader. State: Replica: 990a34a24943459b8220854089c2b29e, State: Running, Role: LEADER
I20260619 02:56:47.329129 27087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990a34a24943459b8220854089c2b29e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35583 } }
I20260619 02:56:47.331743 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "990a34a24943459b8220854089c2b29e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990a34a24943459b8220854089c2b29e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35583 } } }
I20260619 02:56:47.332105 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:47.333570 27089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990a34a24943459b8220854089c2b29e. Latest consensus state: current_term: 1 leader_uuid: "990a34a24943459b8220854089c2b29e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990a34a24943459b8220854089c2b29e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35583 } } }
I20260619 02:56:47.333734 27098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:47.333855 27089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:47.339075 27098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:47.347535 27098 catalog_manager.cc:1347] Generated new cluster ID: c7c2a9a71c8e44378a6051744a864481
I20260619 02:56:47.347654 27098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:47.381840 27098 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:47.383780 27098 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:47.395484 27098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 990a34a24943459b8220854089c2b29e: Generated new TSK 0
I20260619 02:56:47.396374 27098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:47.455976 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35583
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:47.769316 27108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:47.769623 27108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:47.769779 27108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:47.776819 27108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:47.776998 27108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:47.777168 27108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:47.786984 27108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:47.788283 27108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:47.789541 27108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:47.803004 27113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:47.803040 27114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:47.803184 27116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:47.804018 27108 server_base.cc:1034] running on GCE node
I20260619 02:56:47.804579 27108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:47.805434 27108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:47.806672 27108 hybrid_clock.cc:648] HybridClock initialized: now 1781837807806629 us; error 53 us; skew 500 ppm
I20260619 02:56:47.806981 27108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:47 dist-test-slave-2gms krb5kdc[27002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:47.809942 27108 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:47.811394 27108 webserver.cc:460] Webserver started at http://127.22.230.1:34079/ using document root <none> and password file <none>
I20260619 02:56:47.812005 27108 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:47.812125 27108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:47.812364 27108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:47.815285 27108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "12d4dec1650046008ec0971d9eb3b6ec"
format_stamp: "Formatted at 2026-06-19 02:56:47 on dist-test-slave-2gms"
server_key: "9be8085a7c0d15f273a70913ec423556"
server_key_iv: "84bf69e16a949c58815b31770c933781"
server_key_version: "encryptionkey@0"
I20260619 02:56:47.815961 27108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "12d4dec1650046008ec0971d9eb3b6ec"
format_stamp: "Formatted at 2026-06-19 02:56:47 on dist-test-slave-2gms"
server_key: "9be8085a7c0d15f273a70913ec423556"
server_key_iv: "84bf69e16a949c58815b31770c933781"
server_key_version: "encryptionkey@0"
I20260619 02:56:47.820667 27108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:47.824100 27123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:47.825321 27108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:56:47.825502 27108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "12d4dec1650046008ec0971d9eb3b6ec"
format_stamp: "Formatted at 2026-06-19 02:56:47 on dist-test-slave-2gms"
server_key: "9be8085a7c0d15f273a70913ec423556"
server_key_iv: "84bf69e16a949c58815b31770c933781"
server_key_version: "encryptionkey@0"
I20260619 02:56:47.825713 27108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:47.841962 27108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:47.846174 27108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:47.846429 27108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:47.847301 27108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:47.848498 27108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:47.848574 27108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:47.848732 27108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:47.848790 27108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:47.880957 27108 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:35681
I20260619 02:56:47.880998 27235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:35681 every 8 connection(s)
I20260619 02:56:47.882303 27108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:47.883949 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27108
I20260619 02:56:47.884186 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:47.884999 23448 external_mini_cluster.cc:1428] Setting key b1c2227056273fd8598d2339c6681f7c
Jun 19 02:56:47 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837807, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:47.902797 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:47.884753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:42985 (local address 127.22.230.62:35583)
0619 02:56:47.885191 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:47.885196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:47.886427 (+ 1231us) server_negotiation.cc:407] Connection header received
0619 02:56:47.888403 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:47.888412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:47.888497 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:47.888617 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:47.889971 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:47.891246 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:47.892844 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:47.893227 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:47.893319 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:47.897359 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:47.897364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:47.897367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:47.897395 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:47.899857 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:47.900564 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:47.900571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:47.900575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:47.900696 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:47.901113 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:47.901120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:47.901124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:47.901384 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:47.901544 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:47.902192 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 02:56:47.902401 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":241,"threads_started":1}
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:56:47.903746 27239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:47.885425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35583 (local address 127.22.230.1:42985)
0619 02:56:47.886250 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:47.886302 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:47.888140 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:47.888806 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:47.888813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:47.889285 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:47.889809 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:47.889825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:47.891396 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:47.891399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:47.892688 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:47.892703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:47.893437 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:47.893445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:47.893779 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:47.894520 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:47.894552 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:47.897123 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:47.900006 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:47.900014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:47.900028 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:47.900440 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:47.900804 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:47.900810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:47.900814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:47.900995 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:47.901510 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:47.901518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:47.901839 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 02:56:47.902107 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 02:56:47.902621 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":340,"threads_started":1}
I20260619 02:56:47.905288 27236 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35583
I20260619 02:56:47.905606 27236 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:56:47.906363 27236 heartbeater.cc:511] Master 127.22.230.62:35583 requested a full tablet report, sending...
I20260619 02:56:47.908598 27047 ts_manager.cc:194] Registered new tserver with Master: 12d4dec1650046008ec0971d9eb3b6ec (127.22.230.1:35681)
I20260619 02:56:47.910647 27047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:42985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:47.929292 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35583
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:48.237977 27242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:48.238229 27242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:48.238368 27242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:48.245471 27242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:48.245563 27242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:48.245745 27242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:48.254942 27242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:48.256141 27242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:48.257422 27242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:48.269755 27248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:48.269793 27250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:48.269846 27247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:48.270638 27242 server_base.cc:1034] running on GCE node
I20260619 02:56:48.271245 27242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:48.272121 27242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:48.273311 27242 hybrid_clock.cc:648] HybridClock initialized: now 1781837808273282 us; error 29 us; skew 500 ppm
I20260619 02:56:48.273617 27242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:48.276540 27242 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:48.277818 27242 webserver.cc:460] Webserver started at http://127.22.230.2:34577/ using document root <none> and password file <none>
I20260619 02:56:48.278462 27242 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:48.278618 27242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:48.278918 27242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:48.281579 27242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "cf9b57d2e10e419eaf45d58e96157cb2"
format_stamp: "Formatted at 2026-06-19 02:56:48 on dist-test-slave-2gms"
server_key: "780e57d4dcf53c7c53bbbb75b9fb24f9"
server_key_iv: "7c0b49ecae872d280cbbd94defd1315a"
server_key_version: "encryptionkey@0"
I20260619 02:56:48.282310 27242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "cf9b57d2e10e419eaf45d58e96157cb2"
format_stamp: "Formatted at 2026-06-19 02:56:48 on dist-test-slave-2gms"
server_key: "780e57d4dcf53c7c53bbbb75b9fb24f9"
server_key_iv: "7c0b49ecae872d280cbbd94defd1315a"
server_key_version: "encryptionkey@0"
I20260619 02:56:48.287204 27242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:56:48.290386 27257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:48.291666 27242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:56:48.291833 27242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "cf9b57d2e10e419eaf45d58e96157cb2"
format_stamp: "Formatted at 2026-06-19 02:56:48 on dist-test-slave-2gms"
server_key: "780e57d4dcf53c7c53bbbb75b9fb24f9"
server_key_iv: "7c0b49ecae872d280cbbd94defd1315a"
server_key_version: "encryptionkey@0"
I20260619 02:56:48.292021 27242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:48.323933 27242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:48.327737 27242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:48.328012 27242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:48.328960 27242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:48.330219 27242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:48.330310 27242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:48.330454 27242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:48.330539 27242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:48.362296 27242 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:40761
I20260619 02:56:48.362344 27369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:40761 every 8 connection(s)
I20260619 02:56:48.363574 27242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:48.365943 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27242
I20260619 02:56:48.366173 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:48.366668 23448 external_mini_cluster.cc:1428] Setting key 52247dfef6df16567991915f93d10ed3
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:48.385792 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.366079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:35997 (local address 127.22.230.62:35583)
0619 02:56:48.366277 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:48.366283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:48.367692 (+ 1409us) server_negotiation.cc:407] Connection header received
0619 02:56:48.369878 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:48.369882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:48.369940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:48.370018 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:48.372288 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.372973 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.375420 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.376132 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.376276 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.380231 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:48.380236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:48.380239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:48.380268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:48.382921 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.383617 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.383622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.383626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.383758 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.384175 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.384180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.384184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.384433 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:48.384579 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:48.385237 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 02:56:48.385446 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:56:48.386886 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.366733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35583 (local address 127.22.230.2:35997)
0619 02:56:48.367538 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:48.367576 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:48.369587 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:48.370324 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:48.370358 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:48.371196 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:48.372026 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:48.372053 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.373128 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.373136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:48.375291 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:48.375302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.376349 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.376353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:48.376698 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.377404 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:48.377442 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:48.380004 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:48.383070 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:48.383079 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:48.383096 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:48.383491 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:48.383868 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:48.383873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:48.383877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:48.384055 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:48.384555 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:48.384563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:48.384897 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 02:56:48.385154 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 02:56:48.385664 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":357,"threads_started":1}
I20260619 02:56:48.388293 27370 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35583
I20260619 02:56:48.388612 27370 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
I20260619 02:56:48.389469 27370 heartbeater.cc:511] Master 127.22.230.62:35583 requested a full tablet report, sending...
I20260619 02:56:48.390985 27047 ts_manager.cc:194] Registered new tserver with Master: cf9b57d2e10e419eaf45d58e96157cb2 (127.22.230.2:40761)
I20260619 02:56:48.392211 27047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:35997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:48.413403 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35583
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:48.723596 27375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:48.723853 27375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:48.724030 27375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:48.731181 27375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:48.731316 27375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:48.731469 27375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:48.740778 27375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:48.741925 27375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:48.743184 27375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:48.755918 27380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:48.755863 27381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:48.755955 27383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:48.756671 27375 server_base.cc:1034] running on GCE node
I20260619 02:56:48.757280 27375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:48.758153 27375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:48.759369 27375 hybrid_clock.cc:648] HybridClock initialized: now 1781837808759317 us; error 46 us; skew 500 ppm
I20260619 02:56:48.759719 27375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:48.762676 27375 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:48.764053 27375 webserver.cc:460] Webserver started at http://127.22.230.3:39899/ using document root <none> and password file <none>
I20260619 02:56:48.764698 27375 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:48.764838 27375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:48.765141 27375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:48.767941 27375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "64dee96bde12477dbb607a5eeb4fdd20"
format_stamp: "Formatted at 2026-06-19 02:56:48 on dist-test-slave-2gms"
server_key: "668196ce8498886af241db464e6d1067"
server_key_iv: "2787896e71044ba8cd2dcc0665cf885d"
server_key_version: "encryptionkey@0"
I20260619 02:56:48.768682 27375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "64dee96bde12477dbb607a5eeb4fdd20"
format_stamp: "Formatted at 2026-06-19 02:56:48 on dist-test-slave-2gms"
server_key: "668196ce8498886af241db464e6d1067"
server_key_iv: "2787896e71044ba8cd2dcc0665cf885d"
server_key_version: "encryptionkey@0"
I20260619 02:56:48.773622 27375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 02:56:48.776733 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:48.777905 27375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:56:48.778074 27375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "64dee96bde12477dbb607a5eeb4fdd20"
format_stamp: "Formatted at 2026-06-19 02:56:48 on dist-test-slave-2gms"
server_key: "668196ce8498886af241db464e6d1067"
server_key_iv: "2787896e71044ba8cd2dcc0665cf885d"
server_key_version: "encryptionkey@0"
I20260619 02:56:48.778251 27375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:48.810938 27375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:48.815452 27375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:48.815733 27375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:48.816656 27375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:48.817896 27375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:48.818006 27375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:48.818157 27375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:48.818228 27375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:48.850149 27375 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:45573
I20260619 02:56:48.850193 27502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:45573 every 8 connection(s)
I20260619 02:56:48.851528 27375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:48.861374 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27375
I20260619 02:56:48.861570 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:48.862075 23448 external_mini_cluster.cc:1428] Setting key 4cabbce4aeb2a240d86bf16c64473a4d
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:48.870090 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.854070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:42403 (local address 127.22.230.62:35583)
0619 02:56:48.854253 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:48.854257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:48.855458 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 02:56:48.857214 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:48.857218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:48.857279 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:48.857364 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:48.858843 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.859551 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.861024 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.861639 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.861770 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.865324 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:48.865329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:48.865345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:48.865372 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:48.867709 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.868319 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.868323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.868325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.868426 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.868768 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.868771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.868773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.869024 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:48.869125 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:48.869698 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 02:56:48.869839 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:56:48.871050 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.854621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35583 (local address 127.22.230.3:42403)
0619 02:56:48.855312 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:48.855356 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:48.856986 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:48.857581 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:48.857588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:48.858094 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:48.858704 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:48.858723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.859669 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.859673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:48.860852 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:48.860862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.861826 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.861830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:48.862008 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.862720 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:48.862740 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:48.865146 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:48.867822 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:48.867828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:48.867839 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:48.868218 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:48.868509 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:48.868512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:48.868514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:48.868675 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:48.869119 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:48.869124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:48.869387 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 02:56:48.869601 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 02:56:48.869921 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":287,"threads_started":1}
I20260619 02:56:48.872344 27503 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35583
I20260619 02:56:48.872614 27503 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:48.873263 27503 heartbeater.cc:511] Master 127.22.230.62:35583 requested a full tablet report, sending...
I20260619 02:56:48.874641 27047 ts_manager.cc:194] Registered new tserver with Master: 64dee96bde12477dbb607a5eeb4fdd20 (127.22.230.3:45573)
I20260619 02:56:48.875559 27047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:42403
I20260619 02:56:48.877285 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260619 02:56:48.913389 27236 heartbeater.cc:503] Master 127.22.230.62:35583 was elected leader, sending a full tablet report...
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:48.925411 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.911374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.22.230.1:35681)
0619 02:56:48.911977 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:48.911984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:48.912012 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:56:48.912066 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:48.912072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:48.912266 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:48.912387 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:48.913905 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.915252 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.916401 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.916895 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.916985 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.920667 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:48.920677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:48.920691 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:48.920719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:48.923273 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.923826 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.923832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.923834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.923940 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.924252 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.924255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.924257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.924553 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:48.924763 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:48.924955 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:56:48.925131 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":333,"threads_started":1}
Jun 19 02:56:48 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:48.941924 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.929540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.22.230.62:35583)
0619 02:56:48.929754 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:48.929762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:48.929948 (+ 186us) server_negotiation.cc:407] Connection header received
0619 02:56:48.930104 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:48.930108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:48.930164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:48.930253 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:48.931375 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.932052 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.933168 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.933540 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.933625 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.937264 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:48.937268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:48.937271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:48.937299 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:48.939767 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.940386 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.940389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.940391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.940496 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.940788 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.940791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.940792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.941058 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:48.941217 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:48.941460 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 02:56:48.941596 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 02:56:48.942420 27047 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55276
I20260619 02:56:48.943320 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27108
I20260619 02:56:48.954293 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:48.944078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.22.230.1:35681)
0619 02:56:48.944287 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:48.944294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:48.944469 (+ 175us) server_negotiation.cc:407] Connection header received
0619 02:56:48.944585 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:48.944588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:48.944642 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:48.944715 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:48.945716 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.946343 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.947572 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:48.947952 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:48.948043 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:48.950091 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:48.950097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:48.950102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:48.950134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:48.952401 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.952899 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.952903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.952905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.953003 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:48.953280 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:48.953283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:48.953285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:48.953484 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:48.953652 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:48.953840 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 02:56:48.953978 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:56:48.954865 27231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:49.035668 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27108
I20260619 02:56:49.051496 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27242
Jun 19 02:56:49 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:49.066920 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:49.052357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.22.230.2:40761)
0619 02:56:49.052892 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:49.052899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:49.052924 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:56:49.052986 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:49.052993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:49.053211 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:49.053337 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:49.054993 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:49.056127 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:49.057442 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:49.057936 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:49.058023 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:49.061940 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:49.061949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:49.061962 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:49.061990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:49.064516 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:49.065059 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:49.065065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:49.065067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:49.065232 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:49.065550 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:49.065552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:49.065554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:49.065873 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:49.066074 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:49.066329 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 02:56:49.066528 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
I20260619 02:56:49.068104 27365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:49.148969 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27242
I20260619 02:56:49.164853 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27375
Jun 19 02:56:49 dist-test-slave-2gms krb5kdc[27002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:49.180163 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:49.165870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.22.230.3:45573)
0619 02:56:49.166464 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:49.166473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:49.166499 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:56:49.166618 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:49.166628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:49.166844 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:49.167052 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:49.168285 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:49.169290 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:49.170555 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:49.171208 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:49.171344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:49.175380 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:49.175390 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:49.175403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:49.175431 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:49.177911 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:49.178478 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:49.178483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:49.178485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:49.178648 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:49.178962 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:49.178965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:49.178967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:49.179273 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:49.179481 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:49.179699 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 02:56:49.179872 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":304,"threads_started":1}
I20260619 02:56:49.181432 27498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:49.262718 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27375
I20260619 02:56:49.278357 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27015
I20260619 02:56:49.279498 27077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:49.378118 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27015
2026-06-19T02:56:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3784 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:49 dist-test-slave-2gms krb5kdc[27522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:49 dist-test-slave-2gms krb5kdc[27522](info): set up 2 sockets
Jun 19 02:56:49 dist-test-slave-2gms krb5kdc[27522](info): commencing operation
krb5kdc: starting...
W20260619 02:56:50.471022 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:56:50 dist-test-slave-2gms krb5kdc[27522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:56:50.661329 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35383
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:50.963683 27535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:50.963990 27535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:50.964167 27535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:50.971130 27535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:50.971266 27535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:50.971367 27535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:50.971467 27535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:50.971556 27535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:50.980849 27535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:50.982046 27535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:50.983302 27535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:50.994421 27543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:50.994371 27541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:50.994468 27540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:50.995203 27535 server_base.cc:1034] running on GCE node
I20260619 02:56:50.995951 27535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:50.997231 27535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:50.998454 27535 hybrid_clock.cc:648] HybridClock initialized: now 1781837810998425 us; error 30 us; skew 500 ppm
I20260619 02:56:50.998821 27535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:50 dist-test-slave-2gms krb5kdc[27522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837810, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:51.001601 27535 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:56:51.002856 27535 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:51.002980 27535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:51.003289 27535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:51.005957 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "3cc00c6a955d4f329012579d6d9cff78"
format_stamp: "Formatted at 2026-06-19 02:56:51 on dist-test-slave-2gms"
server_key: "d5ff0f99cdd37ad7841649eb083c146a"
server_key_iv: "efab777c5fae2b79ab871eb40c071186"
server_key_version: "encryptionkey@0"
I20260619 02:56:51.006803 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "3cc00c6a955d4f329012579d6d9cff78"
format_stamp: "Formatted at 2026-06-19 02:56:51 on dist-test-slave-2gms"
server_key: "d5ff0f99cdd37ad7841649eb083c146a"
server_key_iv: "efab777c5fae2b79ab871eb40c071186"
server_key_version: "encryptionkey@0"
I20260619 02:56:51.011418 27535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:56:51.014514 27549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:51.015719 27535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:56:51.015895 27535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "3cc00c6a955d4f329012579d6d9cff78"
format_stamp: "Formatted at 2026-06-19 02:56:51 on dist-test-slave-2gms"
server_key: "d5ff0f99cdd37ad7841649eb083c146a"
server_key_iv: "efab777c5fae2b79ab871eb40c071186"
server_key_version: "encryptionkey@0"
I20260619 02:56:51.016108 27535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:51.038614 27535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:51.042847 27535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:51.043097 27535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:51.059621 27535 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:35383
I20260619 02:56:51.059599 27600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:35383 every 8 connection(s)
I20260619 02:56:51.060463 27535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:51.063874 27601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:51.068432 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27535
I20260619 02:56:51.068638 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:51.069226 23448 external_mini_cluster.cc:1428] Setting key ffd525b3e7f950fdae3c63c122163e40
I20260619 02:56:51.074358 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78: Bootstrap starting.
I20260619 02:56:51.077831 27601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:51.079071 27601 log.cc:824] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:51.081900 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78: No bootstrap required, opened a new log
Jun 19 02:56:51 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:51.089985 27601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc00c6a955d4f329012579d6d9cff78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35383 } }
I20260619 02:56:51.090346 27601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:51.090466 27601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc00c6a955d4f329012579d6d9cff78, State: Initialized, Role: FOLLOWER
I20260619 02:56:51.091301 27601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc00c6a955d4f329012579d6d9cff78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35383 } }
I20260619 02:56:51.091531 27601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:51.091717 27601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:51.091900 27601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:51.092312 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:51.071692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.22.230.62:35383)
0619 02:56:51.072535 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:51.072550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:51.072594 (+ 44us) server_negotiation.cc:407] Connection header received
0619 02:56:51.074164 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:51.074183 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:51.074648 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:51.074954 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:51.076773 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:51.078143 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:51.079948 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:51.080667 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:51.080788 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:51.085221 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:51.085234 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:51.085260 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:51.085296 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:51.088322 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:51.089015 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:51.089021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:51.089027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:51.089196 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:51.089557 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:51.089574 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:51.089576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:51.090063 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:51.090369 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:51.090770 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 02:56:51.091211 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":397,"threads_started":1}
I20260619 02:56:51.095156 27601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc00c6a955d4f329012579d6d9cff78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35383 } }
I20260619 02:56:51.095634 27601 leader_election.cc:302] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc00c6a955d4f329012579d6d9cff78; no voters:
I20260619 02:56:51.096210 27601 leader_election.cc:288] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:51.096303 27606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:51.097019 27606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc00c6a955d4f329012579d6d9cff78, State: Running, Role: LEADER
I20260619 02:56:51.097565 27606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc00c6a955d4f329012579d6d9cff78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35383 } }
I20260619 02:56:51.098233 27601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:51.100543 27608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc00c6a955d4f329012579d6d9cff78. Latest consensus state: current_term: 1 leader_uuid: "3cc00c6a955d4f329012579d6d9cff78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc00c6a955d4f329012579d6d9cff78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35383 } } }
I20260619 02:56:51.100518 27607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc00c6a955d4f329012579d6d9cff78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc00c6a955d4f329012579d6d9cff78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35383 } } }
I20260619 02:56:51.101017 27607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:51.101701 27608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:51.103262 27614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:51.108991 27614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:51.117089 27614 catalog_manager.cc:1347] Generated new cluster ID: adc109099e194092a63bfabf4413cb29
I20260619 02:56:51.117194 27614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:51.141486 27614 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:51.142891 27614 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:51.154852 27614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78: Generated new TSK 0
I20260619 02:56:51.155676 27614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:56:51.207342 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35383
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:51.513324 27627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:51.513542 27627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:51.513624 27627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:51.520397 27627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:51.520483 27627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:51.520581 27627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:51.529587 27627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:51.530722 27627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:51.531956 27627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:51.544814 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:51.544834 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:51.545503 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:51.545524 27627 server_base.cc:1034] running on GCE node
I20260619 02:56:51.546193 27627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:51.547057 27627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:51.548245 27627 hybrid_clock.cc:648] HybridClock initialized: now 1781837811548200 us; error 38 us; skew 500 ppm
I20260619 02:56:51.548542 27627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:51 dist-test-slave-2gms krb5kdc[27522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837811, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:51.551395 27627 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:56:51.552469 27627 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:51.552553 27627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:51.552786 27627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:51.555516 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "b0a2dec3f7584a11908bc96fd995245a"
format_stamp: "Formatted at 2026-06-19 02:56:51 on dist-test-slave-2gms"
server_key: "04a9215dc58673d1aad4e131017a549b"
server_key_iv: "ce7cbebcfddd6818d9b548e8bcb2c309"
server_key_version: "encryptionkey@0"
I20260619 02:56:51.556187 27627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "b0a2dec3f7584a11908bc96fd995245a"
format_stamp: "Formatted at 2026-06-19 02:56:51 on dist-test-slave-2gms"
server_key: "04a9215dc58673d1aad4e131017a549b"
server_key_iv: "ce7cbebcfddd6818d9b548e8bcb2c309"
server_key_version: "encryptionkey@0"
I20260619 02:56:51.560501 27627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:51.563712 27641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:51.564904 27627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:56:51.565060 27627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "b0a2dec3f7584a11908bc96fd995245a"
format_stamp: "Formatted at 2026-06-19 02:56:51 on dist-test-slave-2gms"
server_key: "04a9215dc58673d1aad4e131017a549b"
server_key_iv: "ce7cbebcfddd6818d9b548e8bcb2c309"
server_key_version: "encryptionkey@0"
I20260619 02:56:51.565263 27627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:51.589349 27627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:51.593353 27627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:51.593641 27627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:51.594524 27627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:51.595790 27627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:51.595916 27627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:51.596026 27627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:51.596113 27627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:51.627938 27627 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:35031
I20260619 02:56:51.628019 27753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:35031 every 8 connection(s)
I20260619 02:56:51.628760 27627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:51.633627 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27627
I20260619 02:56:51.633836 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:51.634435 23448 external_mini_cluster.cc:1428] Setting key 2e830b77efac59fb80fecb1b2b507eb1
Jun 19 02:56:51 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837811, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:51.648357 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:51.631152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:59289 (local address 127.22.230.62:35383)
0619 02:56:51.631528 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:51.631533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:51.632761 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 02:56:51.634731 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:51.634737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:51.634927 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:51.635094 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:51.636534 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:51.637264 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:51.638867 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:51.639290 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:51.639440 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:51.642921 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:51.642925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:51.642929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:51.642957 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:51.645399 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:51.646099 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:51.646109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:51.646111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:51.646215 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:51.646657 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:51.646662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:51.646665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:51.646909 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:51.647094 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:51.647765 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 02:56:51.647972 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":186,"threads_started":1}
I20260619 02:56:51.649219 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:51.631820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35383 (local address 127.22.230.1:59289)
0619 02:56:51.632584 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:51.632632 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:51.634415 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:51.635318 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:51.635326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:51.635822 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:51.636362 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:51.636379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:51.637412 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:51.637422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:51.638730 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:51.638742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:51.639543 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:51.639550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:51.639773 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:51.640562 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:51.640582 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:51.642727 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:51.645552 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:51.645560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:51.645574 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:51.645975 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:51.646321 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:51.646326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:51.646330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:51.646506 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:51.647076 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:51.647083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:51.647401 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 02:56:51.647664 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 02:56:51.648159 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":333,"threads_started":1}
I20260619 02:56:51.650780 27754 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35383
I20260619 02:56:51.651069 27754 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:51.651654 27754 heartbeater.cc:511] Master 127.22.230.62:35383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:56:51.653666 27566 ts_manager.cc:194] Registered new tserver with Master: b0a2dec3f7584a11908bc96fd995245a (127.22.230.1:35031)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:56:51.655517 27566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:59289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:56:51.677102 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35383
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:51.977722 27760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:51.977982 27760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:51.978143 27760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:51.985050 27760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:51.985179 27760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:51.985381 27760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:51.994782 27760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:51.996083 27760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:51.997435 27760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:52.010058 27765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:52.010098 27766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:52.010036 27768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:52.010993 27760 server_base.cc:1034] running on GCE node
I20260619 02:56:52.011656 27760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:52.012547 27760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:52.013763 27760 hybrid_clock.cc:648] HybridClock initialized: now 1781837812013714 us; error 60 us; skew 500 ppm
I20260619 02:56:52.014061 27760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:52 dist-test-slave-2gms krb5kdc[27522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837812, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:52.017055 27760 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:56:52.018115 27760 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:52.018199 27760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:52.018427 27760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:52.021395 27760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "e6870d9a6a1646c7a1b04c163da2c671"
format_stamp: "Formatted at 2026-06-19 02:56:52 on dist-test-slave-2gms"
server_key: "f38ae0083335bba11b1601cf28ae1f8d"
server_key_iv: "e0263aea95246d58db4fdb3784e30b9a"
server_key_version: "encryptionkey@0"
I20260619 02:56:52.022104 27760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "e6870d9a6a1646c7a1b04c163da2c671"
format_stamp: "Formatted at 2026-06-19 02:56:52 on dist-test-slave-2gms"
server_key: "f38ae0083335bba11b1601cf28ae1f8d"
server_key_iv: "e0263aea95246d58db4fdb3784e30b9a"
server_key_version: "encryptionkey@0"
I20260619 02:56:52.027102 27760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 02:56:52.030297 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:52.031531 27760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:56:52.031697 27760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "e6870d9a6a1646c7a1b04c163da2c671"
format_stamp: "Formatted at 2026-06-19 02:56:52 on dist-test-slave-2gms"
server_key: "f38ae0083335bba11b1601cf28ae1f8d"
server_key_iv: "e0263aea95246d58db4fdb3784e30b9a"
server_key_version: "encryptionkey@0"
I20260619 02:56:52.031896 27760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:52.048071 27760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:52.051756 27760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:52.052014 27760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:52.052922 27760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:52.054152 27760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:52.054260 27760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:52.054380 27760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:52.054473 27760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:52.091030 27760 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:35991
I20260619 02:56:52.091079 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:35991 every 8 connection(s)
I20260619 02:56:52.091931 27760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:52.093537 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27760
I20260619 02:56:52.093734 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:52.094177 23448 external_mini_cluster.cc:1428] Setting key d9a0ca22191f918b313c2be5028435a7
Jun 19 02:56:52 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837812, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:52.111939 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.094390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:37627 (local address 127.22.230.62:35383)
0619 02:56:52.094626 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.094632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.096164 (+ 1532us) server_negotiation.cc:407] Connection header received
0619 02:56:52.097872 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.097880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.097980 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.098090 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:52.099615 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.100735 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.102644 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.103030 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.103123 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.106329 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:52.106334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:52.106338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:52.106366 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:52.109009 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.109752 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.109760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.109765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.109913 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.110288 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.110294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.110298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.110564 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:52.110766 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.111386 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.111605 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:56:52.112924 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.095020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35383 (local address 127.22.230.2:37627)
0619 02:56:52.095996 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.096055 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.097621 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.098322 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.098329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.098878 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:52.099473 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.099495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.100878 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.100887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.102428 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.102444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.103211 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.103216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.103425 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.104079 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:52.104101 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:52.106094 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:52.109184 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:52.109193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:52.109209 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:52.109636 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:52.110006 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:52.110011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:52.110014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:52.110182 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:52.110729 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:52.110737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:52.111050 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.111309 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.111826 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":363,"threads_started":1}
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:56:52.114265 27887 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35383
I20260619 02:56:52.114617 27887 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:52.115335 27887 heartbeater.cc:511] Master 127.22.230.62:35383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
I20260619 02:56:52.116916 27566 ts_manager.cc:194] Registered new tserver with Master: e6870d9a6a1646c7a1b04c163da2c671 (127.22.230.2:35991)
I20260619 02:56:52.118167 27566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:37627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:56:52.139277 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35383
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:56:52.449505 27892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:52.449837 27892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:52.449995 27892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:52.457127 27892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:52.457262 27892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:52.457434 27892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:52.467084 27892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:52.468370 27892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:52.469677 27892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:52.481846 27897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:52.481902 27898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:52.481930 27900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:52.483026 27892 server_base.cc:1034] running on GCE node
I20260619 02:56:52.483566 27892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:52.484444 27892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:52.485661 27892 hybrid_clock.cc:648] HybridClock initialized: now 1781837812485607 us; error 53 us; skew 500 ppm
I20260619 02:56:52.485949 27892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:56:52 dist-test-slave-2gms krb5kdc[27522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837812, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:56:52.493588 27892 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:56:52.494725 27892 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:52.494844 27892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:52.495141 27892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:52.497848 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "fee6dd5b7aa24aa39492a685a33d62db"
format_stamp: "Formatted at 2026-06-19 02:56:52 on dist-test-slave-2gms"
server_key: "86e592ad44f15b1b083728955fb42a90"
server_key_iv: "030c44c19d992f96e3caedd34f233189"
server_key_version: "encryptionkey@0"
I20260619 02:56:52.498725 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "fee6dd5b7aa24aa39492a685a33d62db"
format_stamp: "Formatted at 2026-06-19 02:56:52 on dist-test-slave-2gms"
server_key: "86e592ad44f15b1b083728955fb42a90"
server_key_iv: "030c44c19d992f96e3caedd34f233189"
server_key_version: "encryptionkey@0"
I20260619 02:56:52.503371 27892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:56:52.506524 27906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:52.507809 27892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 02:56:52.507975 27892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "fee6dd5b7aa24aa39492a685a33d62db"
format_stamp: "Formatted at 2026-06-19 02:56:52 on dist-test-slave-2gms"
server_key: "86e592ad44f15b1b083728955fb42a90"
server_key_iv: "030c44c19d992f96e3caedd34f233189"
server_key_version: "encryptionkey@0"
I20260619 02:56:52.508185 27892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:52.527324 27892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:52.530957 27892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:52.531215 27892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:52.532028 27892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:52.533298 27892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:52.533411 27892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:52.533538 27892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:52.533622 27892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:52.567310 27892 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40267
I20260619 02:56:52.567358 28018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40267 every 8 connection(s)
I20260619 02:56:52.568140 27892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:52.576860 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 27892
I20260619 02:56:52.577059 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:52.577524 23448 external_mini_cluster.cc:1428] Setting key accfb8876edb7131221d02bf759e00ba
Jun 19 02:56:52 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837812, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:56:52.587204 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.570640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:35539 (local address 127.22.230.62:35383)
0619 02:56:52.570839 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.570846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.571927 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 02:56:52.573573 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.573577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.573640 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.573728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:52.575140 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.575869 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.577448 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.578076 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.578234 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.581786 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:52.581792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:52.581797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:52.581835 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:52.584389 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.585075 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.585081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.585084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.585196 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.585599 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.585604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.585607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.585855 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:52.586003 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.586630 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.586830 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:56:52.588066 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.571102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35383 (local address 127.22.230.3:35539)
0619 02:56:52.571778 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.571813 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.573344 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.573903 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.573911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.574397 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:52.575001 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.575017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.575993 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.575997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.577313 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.577324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.578308 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.578317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.578643 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.579338 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:56:52.579358 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:52.581603 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:52.584522 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:52.584531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:52.584545 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:52.584956 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:52.585301 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:56:52.585307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:56:52.585310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:56:52.585488 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:56:52.585979 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:52.585987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:52.586281 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.586516 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.587025 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":261,"threads_started":1}
I20260619 02:56:52.589287 28019 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35383
I20260619 02:56:52.589571 28019 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:52.590194 28019 heartbeater.cc:511] Master 127.22.230.62:35383 requested a full tablet report, sending...
I20260619 02:56:52.591703 27566 ts_manager.cc:194] Registered new tserver with Master: fee6dd5b7aa24aa39492a685a33d62db (127.22.230.3:40267)
I20260619 02:56:52.592644 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:56:52.592633 27566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:35539
I20260619 02:56:52.609323 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.597280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.22.230.62:35383)
0619 02:56:52.597428 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.597432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.597893 (+ 461us) server_negotiation.cc:407] Connection header received
0619 02:56:52.598082 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.598093 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.598150 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.598265 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:52.599685 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.600417 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.601653 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.602066 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.602189 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.604338 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:52.604343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:52.604345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:52.604372 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:52.606911 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.607517 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.607524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.607528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.607650 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.608003 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.608009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.608012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.608294 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:52.608520 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.608809 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.609018 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:56:52.613049 27566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:52.619900 27566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:52.642887 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.630962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35991 (local address 127.0.0.1:50472)
0619 02:56:52.631523 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.631557 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.631729 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.636093 (+ 4364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.636100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.636121 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:52.636532 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.636544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.638287 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.638293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.640304 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.640315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.642083 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.642092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.642265 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.642294 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.642376 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.642487 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":20,"thread_start_us":400,"threads_started":1}
I20260619 02:56:52.643076 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.632801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.22.230.3:40267)
0619 02:56:52.634474 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.634486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.634526 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:56:52.634662 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.634673 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.634954 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.635195 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:52.635924 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.637462 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.640705 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.641571 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.641716 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.641936 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.642535 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.642719 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":1337,"threads_started":1}
I20260619 02:56:52.643121 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.633016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:35031 (local address 127.0.0.1:60614)
0619 02:56:52.633779 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.633796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.633905 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.636427 (+ 2522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.636431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.636457 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:52.636853 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.636865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.638386 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.638391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.640561 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.640570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.642255 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.642260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.642556 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.642620 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.642691 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.642813 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":737,"threads_started":1}
I20260619 02:56:52.643525 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.631917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40267 (local address 127.0.0.1:59886)
0619 02:56:52.632471 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.632489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.632600 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.635189 (+ 2589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.635199 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.635237 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:52.635575 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.635591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.637606 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.637612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.640300 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.640314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.641763 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.641769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.641948 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.642007 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.642136 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.642314 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":28,"thread_start_us":314,"threads_started":1}
I20260619 02:56:52.644021 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.633578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.22.230.1:35031)
0619 02:56:52.635868 (+ 2290us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.635875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.635897 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:56:52.635995 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.636001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.636223 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.636430 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:52.637033 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.638179 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.640688 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.642019 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.643257 (+ 1238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.643481 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.643540 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.643695 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":2015,"threads_started":1}
I20260619 02:56:52.644047 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.631149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.22.230.2:35991)
0619 02:56:52.634767 (+ 3618us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.634774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.634795 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:56:52.634927 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.634932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.635167 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.636701 (+ 1534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:52.636918 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.638139 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.640448 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.641838 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.641969 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.642160 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.642518 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.642711 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3472,"thread_start_us":3356,"threads_started":1}
I20260619 02:56:52.647063 27822 tablet_service.cc:1467] Processing CreateTablet for tablet 006b10d8f87a4c258d9e318b132f0358 (DEFAULT_TABLE table=test-table [id=14a31e37bffd41f69822d91ac1dbd6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:52.647401 27954 tablet_service.cc:1467] Processing CreateTablet for tablet 006b10d8f87a4c258d9e318b132f0358 (DEFAULT_TABLE table=test-table [id=14a31e37bffd41f69822d91ac1dbd6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:52.647401 27689 tablet_service.cc:1467] Processing CreateTablet for tablet 006b10d8f87a4c258d9e318b132f0358 (DEFAULT_TABLE table=test-table [id=14a31e37bffd41f69822d91ac1dbd6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:52.648397 27822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006b10d8f87a4c258d9e318b132f0358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:52.648687 27954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006b10d8f87a4c258d9e318b132f0358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:52.648701 27689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006b10d8f87a4c258d9e318b132f0358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:52.658342 27754 heartbeater.cc:503] Master 127.22.230.62:35383 was elected leader, sending a full tablet report...
I20260619 02:56:52.662479 28037 tablet_bootstrap.cc:492] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Bootstrap starting.
I20260619 02:56:52.665539 28037 tablet_bootstrap.cc:654] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:52.666563 28037 log.cc:824] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:52.667318 28038 tablet_bootstrap.cc:492] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Bootstrap starting.
I20260619 02:56:52.668536 28039 tablet_bootstrap.cc:492] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Bootstrap starting.
I20260619 02:56:52.671500 28039 tablet_bootstrap.cc:654] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:52.672240 28038 tablet_bootstrap.cc:654] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:52.672554 28039 log.cc:824] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:52.673640 28038 log.cc:824] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:52.675747 28039 tablet_bootstrap.cc:492] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: No bootstrap required, opened a new log
I20260619 02:56:52.676028 28037 tablet_bootstrap.cc:492] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: No bootstrap required, opened a new log
I20260619 02:56:52.676090 28039 ts_tablet_manager.cc:1397] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 02:56:52.676324 28037 ts_tablet_manager.cc:1397] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260619 02:56:52.676429 28038 tablet_bootstrap.cc:492] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: No bootstrap required, opened a new log
I20260619 02:56:52.676709 28038 ts_tablet_manager.cc:1397] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 02:56:52.683594 28039 raft_consensus.cc:348] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.683925 28039 raft_consensus.cc:374] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:52.684026 28039 raft_consensus.cc:729] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee6dd5b7aa24aa39492a685a33d62db, State: Initialized, Role: FOLLOWER
I20260619 02:56:52.684614 28039 consensus_queue.cc:260] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.685578 28019 heartbeater.cc:503] Master 127.22.230.62:35383 was elected leader, sending a full tablet report...
I20260619 02:56:52.686936 28038 raft_consensus.cc:348] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.686936 28037 raft_consensus.cc:348] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.687345 28037 raft_consensus.cc:374] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:52.687387 28039 ts_tablet_manager.cc:1428] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260619 02:56:52.687412 28038 raft_consensus.cc:374] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:52.687482 28037 raft_consensus.cc:729] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6870d9a6a1646c7a1b04c163da2c671, State: Initialized, Role: FOLLOWER
I20260619 02:56:52.687529 28038 raft_consensus.cc:729] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a2dec3f7584a11908bc96fd995245a, State: Initialized, Role: FOLLOWER
I20260619 02:56:52.688104 28037 consensus_queue.cc:260] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.688104 28038 consensus_queue.cc:260] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.689786 28038 ts_tablet_manager.cc:1428] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260619 02:56:52.690083 27887 heartbeater.cc:503] Master 127.22.230.62:35383 was elected leader, sending a full tablet report...
I20260619 02:56:52.690080 28037 ts_tablet_manager.cc:1428] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 02:56:52.701581 28043 raft_consensus.cc:482] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:52.701814 28043 raft_consensus.cc:504] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.703469 28043 leader_election.cc:288] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6870d9a6a1646c7a1b04c163da2c671 (127.22.230.2:35991), b0a2dec3f7584a11908bc96fd995245a (127.22.230.1:35031)
I20260619 02:56:52.708915 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.703836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:35991 (local address 127.22.230.3:42175)
0619 02:56:52.704172 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.704192 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.704352 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.704646 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.704650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.704683 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:52.704852 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.704860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.705788 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.705793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.707152 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.707164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.708180 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.708186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.708360 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.708382 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.708486 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.708609 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260619 02:56:52.708907 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.703859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:42175 (local address 127.22.230.2:35991)
0619 02:56:52.704042 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.704048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.704282 (+ 234us) server_negotiation.cc:407] Connection header received
0619 02:56:52.704477 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.704482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.704539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.704628 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:52.704974 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.705658 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.707290 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.708027 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.708186 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.708258 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.708506 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.708640 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:56:52.709730 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.703959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:39037 (local address 127.22.230.1:35031)
0619 02:56:52.704174 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.704181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.704741 (+ 560us) server_negotiation.cc:407] Connection header received
0619 02:56:52.704924 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.704930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.704991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.705080 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:56:52.705543 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.706241 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.707890 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.708634 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.708791 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.708873 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.709405 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.709539 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 02:56:52.709827 28046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.703933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:35031 (local address 127.22.230.3:39037)
0619 02:56:52.704633 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:52.704647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:52.704807 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:52.705093 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:52.705097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:52.705119 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:56:52.705408 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.705416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.706384 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.706389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.707742 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:52.707753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.709002 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.709008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:52.709182 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.709225 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 02:56:52.709363 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 02:56:52.709498 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":438,"threads_started":1}
I20260619 02:56:52.710150 27842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b10d8f87a4c258d9e318b132f0358" candidate_uuid: "fee6dd5b7aa24aa39492a685a33d62db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6870d9a6a1646c7a1b04c163da2c671" is_pre_election: true
I20260619 02:56:52.710551 27842 raft_consensus.cc:2454] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee6dd5b7aa24aa39492a685a33d62db in term 0.
I20260619 02:56:52.710688 27709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b10d8f87a4c258d9e318b132f0358" candidate_uuid: "fee6dd5b7aa24aa39492a685a33d62db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a2dec3f7584a11908bc96fd995245a" is_pre_election: true
I20260619 02:56:52.711200 27709 raft_consensus.cc:2454] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee6dd5b7aa24aa39492a685a33d62db in term 0.
I20260619 02:56:52.711253 27908 leader_election.cc:302] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6870d9a6a1646c7a1b04c163da2c671, fee6dd5b7aa24aa39492a685a33d62db; no voters:
I20260619 02:56:52.711633 28043 raft_consensus.cc:2781] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:52.711793 28043 raft_consensus.cc:482] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:52.711912 28043 raft_consensus.cc:3037] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:52.714068 28043 raft_consensus.cc:504] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.714684 28043 leader_election.cc:288] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [CANDIDATE]: Term 1 election: Requested vote from peers e6870d9a6a1646c7a1b04c163da2c671 (127.22.230.2:35991), b0a2dec3f7584a11908bc96fd995245a (127.22.230.1:35031)
I20260619 02:56:52.714967 27842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b10d8f87a4c258d9e318b132f0358" candidate_uuid: "fee6dd5b7aa24aa39492a685a33d62db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6870d9a6a1646c7a1b04c163da2c671"
I20260619 02:56:52.715147 27709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006b10d8f87a4c258d9e318b132f0358" candidate_uuid: "fee6dd5b7aa24aa39492a685a33d62db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a2dec3f7584a11908bc96fd995245a"
I20260619 02:56:52.715175 27842 raft_consensus.cc:3037] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:52.715337 27709 raft_consensus.cc:3037] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:52.717308 27709 raft_consensus.cc:2454] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee6dd5b7aa24aa39492a685a33d62db in term 1.
I20260619 02:56:52.717311 27842 raft_consensus.cc:2454] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee6dd5b7aa24aa39492a685a33d62db in term 1.
I20260619 02:56:52.717679 27909 leader_election.cc:302] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0a2dec3f7584a11908bc96fd995245a, fee6dd5b7aa24aa39492a685a33d62db; no voters:
I20260619 02:56:52.718008 28043 raft_consensus.cc:2781] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:52.718423 28043 raft_consensus.cc:686] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 1 LEADER]: Becoming Leader. State: Replica: fee6dd5b7aa24aa39492a685a33d62db, State: Running, Role: LEADER
I20260619 02:56:52.718935 28043 consensus_queue.cc:237] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } }
I20260619 02:56:52.723135 27564 catalog_manager.cc:5495] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db reported cstate change: term changed from 0 to 1, leader changed from <none> to fee6dd5b7aa24aa39492a685a33d62db (127.22.230.3). New cstate: current_term: 1 leader_uuid: "fee6dd5b7aa24aa39492a685a33d62db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee6dd5b7aa24aa39492a685a33d62db" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 } health_report { overall_health: UNKNOWN } } }
I20260619 02:56:52.739648 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.735085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.22.230.3:40267)
0619 02:56:52.735257 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.735262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.735277 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:56:52.735379 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.735382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.735445 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.735557 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:56:52.735872 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.736660 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.737894 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.738263 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.738346 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.738795 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:56:52.739248 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:56:52.739360 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.739423 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.739476 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:56:52.746990 27842 raft_consensus.cc:1261] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 1 FOLLOWER]: Refusing update from remote peer fee6dd5b7aa24aa39492a685a33d62db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:52.746974 27709 raft_consensus.cc:1261] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 1 FOLLOWER]: Refusing update from remote peer fee6dd5b7aa24aa39492a685a33d62db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:52.747962 28047 consensus_queue.cc:1035] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6870d9a6a1646c7a1b04c163da2c671" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 35991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:52.748303 28043 consensus_queue.cc:1035] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a2dec3f7584a11908bc96fd995245a" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:52.773506 27565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:52.773830 27565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:52.777516 27565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cc00c6a955d4f329012579d6d9cff78: Sending DeleteTablet for 3 replicas of tablet 006b10d8f87a4c258d9e318b132f0358
I20260619 02:56:52.778682 27954 tablet_service.cc:1514] Processing DeleteTablet for tablet 006b10d8f87a4c258d9e318b132f0358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:52 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:59886
I20260619 02:56:52.778818 27689 tablet_service.cc:1514] Processing DeleteTablet for tablet 006b10d8f87a4c258d9e318b132f0358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:52 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:60614
I20260619 02:56:52.779321 27822 tablet_service.cc:1514] Processing DeleteTablet for tablet 006b10d8f87a4c258d9e318b132f0358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:52 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50472
I20260619 02:56:52.780128 28061 tablet_replica.cc:330] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: stopping tablet replica
I20260619 02:56:52.780593 28061 raft_consensus.cc:2229] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:52.781015 28061 raft_consensus.cc:2258] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:52.781689 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27627
I20260619 02:56:52.783111 28061 ts_tablet_manager.cc:1905] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:52.790082 28061 ts_tablet_manager.cc:1918] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:52.790328 28061 log.cc:1196] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/006b10d8f87a4c258d9e318b132f0358
I20260619 02:56:52.790966 28061 ts_tablet_manager.cc:1939] T 006b10d8f87a4c258d9e318b132f0358 P b0a2dec3f7584a11908bc96fd995245a: Deleting consensus metadata
I20260619 02:56:52.791040 28062 tablet_replica.cc:330] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: stopping tablet replica
I20260619 02:56:52.791445 28062 raft_consensus.cc:2229] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:52.791903 28062 raft_consensus.cc:2258] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:52.792935 27552 catalog_manager.cc:4841] TS b0a2dec3f7584a11908bc96fd995245a (127.22.230.1:35031): tablet 006b10d8f87a4c258d9e318b132f0358 (table test-table [id=14a31e37bffd41f69822d91ac1dbd6b8]) successfully deleted
I20260619 02:56:52.794390 28062 ts_tablet_manager.cc:1905] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:52.796027 28060 tablet_replica.cc:330] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: stopping tablet replica
I20260619 02:56:52.796401 28060 raft_consensus.cc:2229] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:52.796995 28060 raft_consensus.cc:2258] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:52.799240 28060 ts_tablet_manager.cc:1905] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:52.801767 28062 ts_tablet_manager.cc:1918] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:52.802155 28062 log.cc:1196] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/006b10d8f87a4c258d9e318b132f0358
I20260619 02:56:52.802762 28062 ts_tablet_manager.cc:1939] T 006b10d8f87a4c258d9e318b132f0358 P e6870d9a6a1646c7a1b04c163da2c671: Deleting consensus metadata
I20260619 02:56:52.804553 27551 catalog_manager.cc:4841] TS e6870d9a6a1646c7a1b04c163da2c671 (127.22.230.2:35991): tablet 006b10d8f87a4c258d9e318b132f0358 (table test-table [id=14a31e37bffd41f69822d91ac1dbd6b8]) successfully deleted
I20260619 02:56:52.806532 28060 ts_tablet_manager.cc:1918] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:52.806735 28060 log.cc:1196] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/006b10d8f87a4c258d9e318b132f0358
I20260619 02:56:52.807302 28060 ts_tablet_manager.cc:1939] T 006b10d8f87a4c258d9e318b132f0358 P fee6dd5b7aa24aa39492a685a33d62db: Deleting consensus metadata
Jun 19 02:56:52 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:56:52.808915 27553 catalog_manager.cc:4841] TS fee6dd5b7aa24aa39492a685a33d62db (127.22.230.3:40267): tablet 006b10d8f87a4c258d9e318b132f0358 (table test-table [id=14a31e37bffd41f69822d91ac1dbd6b8]) successfully deleted
I20260619 02:56:52.814342 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.782976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.22.230.1:35031)
0619 02:56:52.783189 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.783198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.797719 (+ 14521us) server_negotiation.cc:407] Connection header received
0619 02:56:52.798046 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.798051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.798130 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.798204 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:52.799967 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.801096 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.803091 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.803704 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.803803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.808482 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:52.808494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:52.808514 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:52.808553 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:52.811604 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.812316 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.812322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.812325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.812505 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.812857 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.812862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.812865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.813204 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:52.813511 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.813789 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.814002 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:56:52.815016 27749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:52.903129 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27627
I20260619 02:56:52.926443 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27760
Jun 19 02:56:52 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:56:52.942546 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:52.927280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.22.230.2:35991)
0619 02:56:52.927441 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:52.927446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:52.927465 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:56:52.927670 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:52.927674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:52.927744 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:52.927822 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:52.929372 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.930068 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.932068 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:52.932434 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:52.932524 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:52.937227 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:52.937240 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:52.937255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:52.937286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:52.939967 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.940595 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.940602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.940604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.940724 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:52.941088 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:52.941095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:52.941098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:52.941457 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:52.941746 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:52.942012 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 02:56:52.942224 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:56:52.943279 27882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:53.029167 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27760
I20260619 02:56:53.051527 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27892
Jun 19 02:56:53 dist-test-slave-2gms krb5kdc[27522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:56:53.070196 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:53.052369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.22.230.3:40267)
0619 02:56:53.052521 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:53.052525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:53.052554 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:56:53.052682 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:53.052685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:53.052741 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:53.052822 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:53.054442 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:53.055312 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:53.057110 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:53.057537 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:53.057617 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:53.062993 (+ 5376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:53.063008 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:53.063027 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:56:53.063071 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:53.067203 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:53.067993 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:53.068002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:53.068006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:53.068179 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:56:53.068559 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:56:53.068564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:56:53.068568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:56:53.068990 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:53.069333 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:53.069557 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:56:53.069809 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:56:53.071123 28014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:53.186684 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27892
I20260619 02:56:53.205677 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 27535
I20260619 02:56:53.206786 27596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:53.311890 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 27535
2026-06-19T02:56:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3933 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T02:56:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:53Z Disabled control of system clock
I20260619 02:56:53.376633 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:34857
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:34857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:56:53.683712 28071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:53.683966 28071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:53.684056 28071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:53.690883 28071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:53.690971 28071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:53.691013 28071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:53.691066 28071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:53.691107 28071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:56:53.699641 28071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:34857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:34857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:53.700711 28071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:53.702409 28071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:53.713825 28076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:53.713802 28077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:53.714054 28079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:53.714530 28071 server_base.cc:1034] running on GCE node
I20260619 02:56:53.715266 28071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:53.716508 28071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:53.717784 28071 hybrid_clock.cc:648] HybridClock initialized: now 1781837813717752 us; error 32 us; skew 500 ppm
I20260619 02:56:53.718104 28071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:53.720201 28071 webserver.cc:460] Webserver started at http://127.22.230.62:34139/ using document root <none> and password file <none>
I20260619 02:56:53.720808 28071 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:53.720889 28071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:53.721159 28071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:53.723898 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "3df5b94d0d0341ed8f04929b880f957e"
format_stamp: "Formatted at 2026-06-19 02:56:53 on dist-test-slave-2gms"
server_key: "daefdfb54841a53ab47491fa66ed4e00"
server_key_iv: "e30113440e5f99634d1e730439aa76f4"
server_key_version: "encryptionkey@0"
I20260619 02:56:53.724632 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "3df5b94d0d0341ed8f04929b880f957e"
format_stamp: "Formatted at 2026-06-19 02:56:53 on dist-test-slave-2gms"
server_key: "daefdfb54841a53ab47491fa66ed4e00"
server_key_iv: "e30113440e5f99634d1e730439aa76f4"
server_key_version: "encryptionkey@0"
I20260619 02:56:53.729233 28071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:56:53.732375 28085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:53.733579 28071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:56:53.733739 28071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "3df5b94d0d0341ed8f04929b880f957e"
format_stamp: "Formatted at 2026-06-19 02:56:53 on dist-test-slave-2gms"
server_key: "daefdfb54841a53ab47491fa66ed4e00"
server_key_iv: "e30113440e5f99634d1e730439aa76f4"
server_key_version: "encryptionkey@0"
I20260619 02:56:53.733942 28071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:53.739434 28071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:53.740119 28071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:53.740392 28071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:53.757719 28071 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:34857
I20260619 02:56:53.757756 28136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:34857 every 8 connection(s)
I20260619 02:56:53.758924 28071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:53.762512 28137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:53.762696 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28071
I20260619 02:56:53.762928 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:53.763576 23448 external_mini_cluster.cc:1428] Setting key f0c5f59f626b8f109e5ebbd04cc7642a
I20260619 02:56:53.771395 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:53.765436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.22.230.62:34857)
0619 02:56:53.766441 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:53.766461 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:53.766501 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:56:53.768132 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:53.768152 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:53.768555 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:53.768856 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:53.769921 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:53.769926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:53.769938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:53.769953 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:53.770009 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:53.770024 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:53.770181 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:53.770309 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 02:56:53.770554 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":369,"threads_started":1}
I20260619 02:56:53.774348 28137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e: Bootstrap starting.
I20260619 02:56:53.777026 28137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:53.777926 28137 log.cc:824] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:53.780303 28137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e: No bootstrap required, opened a new log
I20260619 02:56:53.786844 28137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df5b94d0d0341ed8f04929b880f957e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34857 } }
I20260619 02:56:53.787189 28137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:53.787289 28137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df5b94d0d0341ed8f04929b880f957e, State: Initialized, Role: FOLLOWER
I20260619 02:56:53.787806 28137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df5b94d0d0341ed8f04929b880f957e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34857 } }
I20260619 02:56:53.788021 28137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:53.788160 28137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:53.788327 28137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:53.790236 28137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df5b94d0d0341ed8f04929b880f957e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34857 } }
I20260619 02:56:53.790756 28137 leader_election.cc:302] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3df5b94d0d0341ed8f04929b880f957e; no voters:
I20260619 02:56:53.791280 28137 leader_election.cc:288] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:53.791379 28142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:53.792229 28142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [term 1 LEADER]: Becoming Leader. State: Replica: 3df5b94d0d0341ed8f04929b880f957e, State: Running, Role: LEADER
I20260619 02:56:53.792747 28142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df5b94d0d0341ed8f04929b880f957e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34857 } }
I20260619 02:56:53.793300 28137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:53.795508 28143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3df5b94d0d0341ed8f04929b880f957e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df5b94d0d0341ed8f04929b880f957e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34857 } } }
I20260619 02:56:53.795871 28143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:53.796640 28144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3df5b94d0d0341ed8f04929b880f957e. Latest consensus state: current_term: 1 leader_uuid: "3df5b94d0d0341ed8f04929b880f957e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3df5b94d0d0341ed8f04929b880f957e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34857 } } }
I20260619 02:56:53.796983 28144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:53.799427 28150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:53.803746 28150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:53.810910 28150 catalog_manager.cc:1347] Generated new cluster ID: 277c63da97dd42aeb13975459041dda4
I20260619 02:56:53.811012 28150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:53.829779 28150 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:53.830215 28150 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:53.838716 28150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e: Generated new TSK 0
I20260619 02:56:53.839409 28150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:56:53.855829 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34857
--builtin_ntp_servers=127.22.230.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 02:56:54.169801 28161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:54.170064 28161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:54.170219 28161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:54.177076 28161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:54.177157 28161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:54.177320 28161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:56:54.185678 28161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:54.186872 28161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:54.188565 28161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:54.201547 28169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:54.201622 28167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:54.201576 28166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:54.202322 28161 server_base.cc:1034] running on GCE node
I20260619 02:56:54.202922 28161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:54.203763 28161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:54.204975 28161 hybrid_clock.cc:648] HybridClock initialized: now 1781837814204928 us; error 58 us; skew 500 ppm
I20260619 02:56:54.205278 28161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:54.207402 28161 webserver.cc:460] Webserver started at http://127.22.230.1:35625/ using document root <none> and password file <none>
I20260619 02:56:54.208035 28161 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:54.208168 28161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:54.208487 28161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:54.211234 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435"
format_stamp: "Formatted at 2026-06-19 02:56:54 on dist-test-slave-2gms"
server_key: "2b93c8f5e3076fc0ede4fead6a3ab4dc"
server_key_iv: "afb5e038ee356a2b6f11c97a900c9a02"
server_key_version: "encryptionkey@0"
I20260619 02:56:54.211992 28161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435"
format_stamp: "Formatted at 2026-06-19 02:56:54 on dist-test-slave-2gms"
server_key: "2b93c8f5e3076fc0ede4fead6a3ab4dc"
server_key_iv: "afb5e038ee356a2b6f11c97a900c9a02"
server_key_version: "encryptionkey@0"
I20260619 02:56:54.216732 28161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:54.219826 28175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:54.221040 28161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:56:54.221215 28161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435"
format_stamp: "Formatted at 2026-06-19 02:56:54 on dist-test-slave-2gms"
server_key: "2b93c8f5e3076fc0ede4fead6a3ab4dc"
server_key_iv: "afb5e038ee356a2b6f11c97a900c9a02"
server_key_version: "encryptionkey@0"
I20260619 02:56:54.221421 28161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:54.226715 28161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:54.227442 28161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:54.227698 28161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:54.228550 28161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:54.229791 28161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:54.229923 28161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:54.230043 28161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:54.230129 28161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:54.261904 28161 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40863
I20260619 02:56:54.261955 28287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40863 every 8 connection(s)
I20260619 02:56:54.263249 28161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:56:54.266989 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28161
I20260619 02:56:54.267176 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:56:54.267793 23448 external_mini_cluster.cc:1428] Setting key 01b9e2dfc92d45eac7ced48740109ef6
I20260619 02:56:54.270493 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34857
--builtin_ntp_servers=127.22.230.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:56:54.271442 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:54.265623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:42829 (local address 127.22.230.62:34857)
0619 02:56:54.265782 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:54.265787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:54.267266 (+ 1479us) server_negotiation.cc:407] Connection header received
0619 02:56:54.269115 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:54.269120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:54.269169 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:54.269280 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:54.270585 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:54.270594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:54.270599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:54.270617 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:54.270658 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:54.270710 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:54.270827 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:54.271041 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 02:56:54.271104 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:56:54.272305 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:54.266136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34857 (local address 127.22.230.1:42829)
0619 02:56:54.267089 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:54.267134 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:54.268865 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:54.269531 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:54.269538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:54.269621 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:54.270269 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:54.270271 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:54.270311 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:54.270387 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:54.270827 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:54.270838 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:54.270848 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 02:56:54.271086 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 02:56:54.271305 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":301,"threads_started":1}
I20260619 02:56:54.273751 28288 heartbeater.cc:347] Connected to a master server at 127.22.230.62:34857
I20260619 02:56:54.274008 28288 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:54.274590 28288 heartbeater.cc:511] Master 127.22.230.62:34857 requested a full tablet report, sending...
I20260619 02:56:54.276710 28102 ts_manager.cc:194] Registered new tserver with Master: 3bcd806f194f4fc1b7cf6c6bdd4db435 (127.22.230.1:40863)
W20260619 02:56:54.581583 28291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:54.581861 28291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:54.582018 28291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:54.589282 28291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:54.589395 28291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:54.589596 28291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:56:54.598273 28291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:54.599481 28291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:54.601109 28291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:54.613538 28296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:54.613759 28297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:54.614001 28299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:54.614477 28291 server_base.cc:1034] running on GCE node
I20260619 02:56:54.615049 28291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:54.615842 28291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:54.617089 28291 hybrid_clock.cc:648] HybridClock initialized: now 1781837814617047 us; error 27 us; skew 500 ppm
I20260619 02:56:54.617425 28291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:54.619488 28291 webserver.cc:460] Webserver started at http://127.22.230.2:33317/ using document root <none> and password file <none>
I20260619 02:56:54.620092 28291 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:54.620198 28291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:54.620528 28291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:54.623092 28291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "78aa2a60e563479a89bb03f635e043e2"
format_stamp: "Formatted at 2026-06-19 02:56:54 on dist-test-slave-2gms"
server_key: "8a31f07c33360104504d5b561494ac37"
server_key_iv: "cab30f9fa9e5ea59d1b9c1c9be92169d"
server_key_version: "encryptionkey@0"
I20260619 02:56:54.623801 28291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "78aa2a60e563479a89bb03f635e043e2"
format_stamp: "Formatted at 2026-06-19 02:56:54 on dist-test-slave-2gms"
server_key: "8a31f07c33360104504d5b561494ac37"
server_key_iv: "cab30f9fa9e5ea59d1b9c1c9be92169d"
server_key_version: "encryptionkey@0"
I20260619 02:56:54.628535 28291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 02:56:54.631599 28305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:54.632794 28291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:56:54.632956 28291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "78aa2a60e563479a89bb03f635e043e2"
format_stamp: "Formatted at 2026-06-19 02:56:54 on dist-test-slave-2gms"
server_key: "8a31f07c33360104504d5b561494ac37"
server_key_iv: "cab30f9fa9e5ea59d1b9c1c9be92169d"
server_key_version: "encryptionkey@0"
I20260619 02:56:54.633164 28291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:54.638427 28291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:54.639150 28291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:54.639392 28291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:54.640184 28291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:54.641433 28291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:54.641566 28291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:54.641690 28291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:54.641769 28291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:54.673098 28291 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:44571
I20260619 02:56:54.673146 28417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:44571 every 8 connection(s)
I20260619 02:56:54.674310 28291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:56:54.680068 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28291
I20260619 02:56:54.680250 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:56:54.680682 23448 external_mini_cluster.cc:1428] Setting key a01bda56191c2b2e7a67717c3ebe861d
I20260619 02:56:54.682739 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:54.676825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:37299 (local address 127.22.230.62:34857)
0619 02:56:54.676986 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:54.676990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:54.678266 (+ 1276us) server_negotiation.cc:407] Connection header received
0619 02:56:54.679874 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:54.679882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:54.679947 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:54.680065 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:54.681446 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:54.681453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:54.681458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:54.681475 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:54.681511 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:54.681526 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:54.681657 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:54.682385 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 02:56:54.682443 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:56:54.683296 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34857
--builtin_ntp_servers=127.22.230.20:35643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 02:56:54.683678 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:54.677335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34857 (local address 127.22.230.2:37299)
0619 02:56:54.678146 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:54.678187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:54.679625 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:54.680319 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:54.680330 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:54.680425 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:54.681093 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:54.681095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:54.681155 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:54.681265 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:54.682114 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:54.682122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:54.682128 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:56:54.682443 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 02:56:54.682679 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":257,"threads_started":1}
I20260619 02:56:54.684934 28418 heartbeater.cc:347] Connected to a master server at 127.22.230.62:34857
I20260619 02:56:54.685207 28418 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:54.685739 28418 heartbeater.cc:511] Master 127.22.230.62:34857 requested a full tablet report, sending...
I20260619 02:56:54.687109 28102 ts_manager.cc:194] Registered new tserver with Master: 78aa2a60e563479a89bb03f635e043e2 (127.22.230.2:44571)
W20260619 02:56:54.991986 28421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:54.992236 28421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:54.992352 28421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:54.999588 28421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:54.999683 28421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:54.999784 28421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:56:55.008252 28421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:34857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:55.009307 28421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:55.011098 28421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:55.023406 28429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:55.023434 28427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:55.023954 28426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:55.024224 28421 server_base.cc:1034] running on GCE node
I20260619 02:56:55.024797 28421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:55.025730 28421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:55.026952 28421 hybrid_clock.cc:648] HybridClock initialized: now 1781837815026903 us; error 51 us; skew 500 ppm
I20260619 02:56:55.027245 28421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:55.029238 28421 webserver.cc:460] Webserver started at http://127.22.230.3:46375/ using document root <none> and password file <none>
I20260619 02:56:55.029881 28421 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:55.030002 28421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:55.030325 28421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:55.032969 28421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "31c47467a6e0446d97d8b314ea054025"
format_stamp: "Formatted at 2026-06-19 02:56:55 on dist-test-slave-2gms"
server_key: "876bc43b4d27255e275556f08d177073"
server_key_iv: "490a34c74f5c53110e392d978bbadd2c"
server_key_version: "encryptionkey@0"
I20260619 02:56:55.033706 28421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "31c47467a6e0446d97d8b314ea054025"
format_stamp: "Formatted at 2026-06-19 02:56:55 on dist-test-slave-2gms"
server_key: "876bc43b4d27255e275556f08d177073"
server_key_iv: "490a34c74f5c53110e392d978bbadd2c"
server_key_version: "encryptionkey@0"
I20260619 02:56:55.038841 28421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 02:56:55.042097 28435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:55.043334 28421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:56:55.043500 28421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "31c47467a6e0446d97d8b314ea054025"
format_stamp: "Formatted at 2026-06-19 02:56:55 on dist-test-slave-2gms"
server_key: "876bc43b4d27255e275556f08d177073"
server_key_iv: "490a34c74f5c53110e392d978bbadd2c"
server_key_version: "encryptionkey@0"
I20260619 02:56:55.043709 28421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:55.049033 28421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:55.049754 28421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:55.050005 28421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:55.051648 28421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:56:55.052945 28421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:56:55.053061 28421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:55.053184 28421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:56:55.053280 28421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:55.084297 28421 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:39103
I20260619 02:56:55.084350 28547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:39103 every 8 connection(s)
I20260619 02:56:55.085559 28421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:56:55.091190 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28421
I20260619 02:56:55.091357 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:56:55.091789 23448 external_mini_cluster.cc:1428] Setting key ad41ee11670d0f740d7f7cdaa73d5a59
I20260619 02:56:55.093513 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.088016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:49877 (local address 127.22.230.62:34857)
0619 02:56:55.088207 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.088212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.089517 (+ 1305us) server_negotiation.cc:407] Connection header received
0619 02:56:55.091229 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.091234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.091300 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.091392 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.092777 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.092783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.092788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.092809 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.092845 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.092857 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.092975 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.093208 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.093271 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:56:55.094652 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.088515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34857 (local address 127.22.230.3:49877)
0619 02:56:55.089365 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:55.089405 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:55.090959 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:55.091604 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:55.091612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:55.091707 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:55.092450 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:55.092454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:55.092503 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:55.092612 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:55.092954 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:55.092965 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:55.092972 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:56:55.093304 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 02:56:55.093539 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":251,"threads_started":1}
I20260619 02:56:55.095803 28548 heartbeater.cc:347] Connected to a master server at 127.22.230.62:34857
I20260619 02:56:55.096079 28548 heartbeater.cc:464] Registering TS with master...
I20260619 02:56:55.096617 28548 heartbeater.cc:511] Master 127.22.230.62:34857 requested a full tablet report, sending...
I20260619 02:56:55.097843 28102 ts_manager.cc:194] Registered new tserver with Master: 31c47467a6e0446d97d8b314ea054025 (127.22.230.3:39103)
I20260619 02:56:55.106663 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:56:55.113179 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.110955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.22.230.62:34857)
0619 02:56:55.111133 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.111142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.111596 (+ 454us) server_negotiation.cc:407] Connection header received
0619 02:56:55.111715 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.111718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.111765 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.111833 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.112672 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.112676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.112679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.112690 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.112713 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.112723 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.112806 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.112839 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.112889 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:56:55.116775 28102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:56:55.123266 28102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:56:55.155511 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.134375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.22.230.1:40863)
0619 02:56:55.151666 (+ 17291us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.151677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.151707 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:56:55.151776 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.151783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.152034 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.152200 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.154641 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.154652 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.154668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.154693 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.154750 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.154770 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.154892 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.155095 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.155183 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17047,"thread_start_us":17547,"threads_started":1}
I20260619 02:56:55.155550 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.134650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.22.230.3:39103)
0619 02:56:55.150606 (+ 15956us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.150619 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.152708 (+ 2089us) server_negotiation.cc:407] Connection header received
0619 02:56:55.152782 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.152789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.152981 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.153131 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.154630 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.154638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.154653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.154682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.154746 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.154768 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.154956 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.155057 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.155150 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15716,"thread_start_us":15532,"threads_started":1}
I20260619 02:56:55.155579 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.134260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:39103 (local address 127.0.0.1:54348)
0619 02:56:55.152482 (+ 18222us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:55.152503 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:55.152596 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:55.153446 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:55.153453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:55.153476 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:55.154269 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:55.154274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:55.154314 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:55.154427 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:55.154882 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:55.154890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:55.154898 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:56:55.155078 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 02:56:55.155154 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18103,"thread_start_us":18739,"threads_started":1}
I20260619 02:56:55.156230 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.134139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40863 (local address 127.0.0.1:35028)
0619 02:56:55.150947 (+ 16808us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:55.150997 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:55.151134 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:55.153198 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:55.153206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:55.153294 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:55.154356 (+ 1062us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:55.154361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:55.154378 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:55.154436 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:55.154940 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:55.154944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:55.154949 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:56:55.155754 (+ 805us) client_negotiation.cc:240] Negotiation successful
0619 02:56:55.155801 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16568,"thread_start_us":18078,"threads_started":1}
I20260619 02:56:55.157132 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.136626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.22.230.2:44571)
0619 02:56:55.152842 (+ 16216us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.152852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.153339 (+ 487us) server_negotiation.cc:407] Connection header received
0619 02:56:55.153557 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.153566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.153757 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.153872 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.156338 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.156346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.156357 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.156376 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.156433 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.156464 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.156621 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.156748 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.156829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16000,"thread_start_us":15890,"threads_started":1}
I20260619 02:56:55.157141 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.136415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:44571 (local address 127.0.0.1:51170)
0619 02:56:55.153224 (+ 16809us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:55.153241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:55.153354 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:55.154167 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:55.154172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:55.154194 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:55.156044 (+ 1850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:55.156050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:55.156066 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:55.156182 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:55.156633 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:55.156640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:55.156646 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:56:55.156745 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:56:55.156786 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16704,"thread_start_us":14146,"threads_started":1}
I20260619 02:56:55.159801 28483 tablet_service.cc:1467] Processing CreateTablet for tablet cf2828d50af341b29111cbc514825aea (DEFAULT_TABLE table=test-table [id=62711fb31f634ea58f4cd01a06a65cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:55.159832 28223 tablet_service.cc:1467] Processing CreateTablet for tablet cf2828d50af341b29111cbc514825aea (DEFAULT_TABLE table=test-table [id=62711fb31f634ea58f4cd01a06a65cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:55.160392 28353 tablet_service.cc:1467] Processing CreateTablet for tablet cf2828d50af341b29111cbc514825aea (DEFAULT_TABLE table=test-table [id=62711fb31f634ea58f4cd01a06a65cc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:56:55.160976 28483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2828d50af341b29111cbc514825aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:55.161072 28223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2828d50af341b29111cbc514825aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:55.161491 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2828d50af341b29111cbc514825aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:55.171619 28566 tablet_bootstrap.cc:492] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Bootstrap starting.
I20260619 02:56:55.174678 28566 tablet_bootstrap.cc:654] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:55.174937 28568 tablet_bootstrap.cc:492] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Bootstrap starting.
I20260619 02:56:55.175647 28567 tablet_bootstrap.cc:492] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Bootstrap starting.
I20260619 02:56:55.176184 28566 log.cc:824] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:55.179124 28567 tablet_bootstrap.cc:654] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:55.179425 28568 tablet_bootstrap.cc:654] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:55.180083 28567 log.cc:824] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:55.180544 28568 log.cc:824] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:55.181229 28566 tablet_bootstrap.cc:492] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: No bootstrap required, opened a new log
I20260619 02:56:55.181535 28566 ts_tablet_manager.cc:1397] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 02:56:55.182349 28567 tablet_bootstrap.cc:492] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: No bootstrap required, opened a new log
I20260619 02:56:55.182637 28567 ts_tablet_manager.cc:1397] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 02:56:55.184767 28568 tablet_bootstrap.cc:492] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: No bootstrap required, opened a new log
I20260619 02:56:55.185041 28568 ts_tablet_manager.cc:1397] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260619 02:56:55.189667 28567 raft_consensus.cc:348] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.190006 28567 raft_consensus.cc:374] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:55.190126 28567 raft_consensus.cc:729] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bcd806f194f4fc1b7cf6c6bdd4db435, State: Initialized, Role: FOLLOWER
I20260619 02:56:55.190763 28567 consensus_queue.cc:260] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.191742 28288 heartbeater.cc:503] Master 127.22.230.62:34857 was elected leader, sending a full tablet report...
I20260619 02:56:55.191751 28566 raft_consensus.cc:348] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.192108 28566 raft_consensus.cc:374] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:55.192188 28566 raft_consensus.cc:729] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c47467a6e0446d97d8b314ea054025, State: Initialized, Role: FOLLOWER
I20260619 02:56:55.192301 28567 ts_tablet_manager.cc:1428] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260619 02:56:55.192768 28566 consensus_queue.cc:260] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.193789 28548 heartbeater.cc:503] Master 127.22.230.62:34857 was elected leader, sending a full tablet report...
I20260619 02:56:55.194344 28566 ts_tablet_manager.cc:1428] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260619 02:56:55.196027 28568 raft_consensus.cc:348] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.196450 28568 raft_consensus.cc:374] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:55.196586 28568 raft_consensus.cc:729] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78aa2a60e563479a89bb03f635e043e2, State: Initialized, Role: FOLLOWER
I20260619 02:56:55.197152 28568 consensus_queue.cc:260] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.198122 28418 heartbeater.cc:503] Master 127.22.230.62:34857 was elected leader, sending a full tablet report...
I20260619 02:56:55.198652 28568 ts_tablet_manager.cc:1428] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260619 02:56:55.223341 28574 raft_consensus.cc:482] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:56:55.223572 28574 raft_consensus.cc:504] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.224959 28574 leader_election.cc:288] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bcd806f194f4fc1b7cf6c6bdd4db435 (127.22.230.1:40863), 31c47467a6e0446d97d8b314ea054025 (127.22.230.3:39103)
I20260619 02:56:55.228091 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.225585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:43331 (local address 127.22.230.3:39103)
0619 02:56:55.225860 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.225868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.226453 (+ 585us) server_negotiation.cc:407] Connection header received
0619 02:56:55.226614 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.226619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.226671 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.226739 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.227511 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.227516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.227521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.227548 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.227578 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.227596 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.227707 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.227796 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.227850 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 02:56:55.228106 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.225183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:36663 (local address 127.22.230.1:40863)
0619 02:56:55.225386 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.225394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.226141 (+ 747us) server_negotiation.cc:407] Connection header received
0619 02:56:55.226319 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.226324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.226375 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.226473 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.227487 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.227492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.227496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.227513 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.227546 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.227562 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.227678 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.227805 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.227852 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:56:55.228377 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.225280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40863 (local address 127.22.230.2:36663)
0619 02:56:55.226040 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:55.226056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:55.226188 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:55.226518 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:55.226525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:55.226548 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:55.227253 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:55.227257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:55.227276 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:55.227329 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:55.227681 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:55.227685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:55.227690 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:56:55.227822 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:56:55.227852 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":444,"threads_started":1}
I20260619 02:56:55.228487 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.225375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:39103 (local address 127.22.230.2:43331)
0619 02:56:55.226354 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:55.226370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:55.226476 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:55.226808 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:55.226813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:55.226834 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:56:55.227339 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:56:55.227341 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:56:55.227353 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:56:55.227388 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:56:55.227685 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:56:55.227689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:56:55.227693 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 02:56:55.227794 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 02:56:55.227847 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":697,"threads_started":1}
I20260619 02:56:55.229339 28243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2828d50af341b29111cbc514825aea" candidate_uuid: "78aa2a60e563479a89bb03f635e043e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" is_pre_election: true
I20260619 02:56:55.229391 28503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2828d50af341b29111cbc514825aea" candidate_uuid: "78aa2a60e563479a89bb03f635e043e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c47467a6e0446d97d8b314ea054025" is_pre_election: true
I20260619 02:56:55.229863 28503 raft_consensus.cc:2454] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78aa2a60e563479a89bb03f635e043e2 in term 0.
I20260619 02:56:55.229863 28243 raft_consensus.cc:2454] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78aa2a60e563479a89bb03f635e043e2 in term 0.
I20260619 02:56:55.230475 28306 leader_election.cc:302] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bcd806f194f4fc1b7cf6c6bdd4db435, 78aa2a60e563479a89bb03f635e043e2; no voters:
I20260619 02:56:55.230845 28574 raft_consensus.cc:2781] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:56:55.230983 28574 raft_consensus.cc:482] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:56:55.231088 28574 raft_consensus.cc:3037] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:55.233215 28574 raft_consensus.cc:504] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.233712 28574 leader_election.cc:288] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [CANDIDATE]: Term 1 election: Requested vote from peers 3bcd806f194f4fc1b7cf6c6bdd4db435 (127.22.230.1:40863), 31c47467a6e0446d97d8b314ea054025 (127.22.230.3:39103)
I20260619 02:56:55.234158 28243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2828d50af341b29111cbc514825aea" candidate_uuid: "78aa2a60e563479a89bb03f635e043e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435"
I20260619 02:56:55.234205 28503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2828d50af341b29111cbc514825aea" candidate_uuid: "78aa2a60e563479a89bb03f635e043e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c47467a6e0446d97d8b314ea054025"
I20260619 02:56:55.234372 28243 raft_consensus.cc:3037] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:55.234392 28503 raft_consensus.cc:3037] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:55.236887 28503 raft_consensus.cc:2454] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78aa2a60e563479a89bb03f635e043e2 in term 1.
I20260619 02:56:55.236913 28243 raft_consensus.cc:2454] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78aa2a60e563479a89bb03f635e043e2 in term 1.
I20260619 02:56:55.237275 28309 leader_election.cc:302] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c47467a6e0446d97d8b314ea054025, 78aa2a60e563479a89bb03f635e043e2; no voters:
I20260619 02:56:55.237577 28574 raft_consensus.cc:2781] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:55.238060 28574 raft_consensus.cc:686] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 1 LEADER]: Becoming Leader. State: Replica: 78aa2a60e563479a89bb03f635e043e2, State: Running, Role: LEADER
I20260619 02:56:55.238478 28574 consensus_queue.cc:237] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } }
I20260619 02:56:55.243229 28101 catalog_manager.cc:5495] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78aa2a60e563479a89bb03f635e043e2 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "78aa2a60e563479a89bb03f635e043e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78aa2a60e563479a89bb03f635e043e2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44571 } health_report { overall_health: HEALTHY } } }
W20260619 02:56:55.265290 28289 tablet.cc:2369] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:56:55.275166 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.273146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.22.230.2:44571)
0619 02:56:55.273285 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.273289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.273470 (+ 181us) server_negotiation.cc:407] Connection header received
0619 02:56:55.273624 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.273627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.273707 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.273779 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.274749 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.274753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.274756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.274769 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.274797 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.274808 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.274903 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.274964 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.275003 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:56:55.282073 28243 raft_consensus.cc:1261] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 1 FOLLOWER]: Refusing update from remote peer 78aa2a60e563479a89bb03f635e043e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:55.282081 28503 raft_consensus.cc:1261] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 1 FOLLOWER]: Refusing update from remote peer 78aa2a60e563479a89bb03f635e043e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:56:55.282969 28577 consensus_queue.cc:1035] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bcd806f194f4fc1b7cf6c6bdd4db435" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:55.283274 28574 consensus_queue.cc:1035] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c47467a6e0446d97d8b314ea054025" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:56:55.306455 28101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:55.306761 28101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:56:55.309927 28101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3df5b94d0d0341ed8f04929b880f957e: Sending DeleteTablet for 3 replicas of tablet cf2828d50af341b29111cbc514825aea
I20260619 02:56:55.310904 28223 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2828d50af341b29111cbc514825aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:55 UTC) from {username='slave'} at 127.0.0.1:35028
I20260619 02:56:55.310930 28483 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2828d50af341b29111cbc514825aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:55 UTC) from {username='slave'} at 127.0.0.1:54348
I20260619 02:56:55.311568 28353 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2828d50af341b29111cbc514825aea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:56:55 UTC) from {username='slave'} at 127.0.0.1:51170
I20260619 02:56:55.311705 28589 tablet_replica.cc:330] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: stopping tablet replica
I20260619 02:56:55.312065 28589 raft_consensus.cc:2229] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:55.312389 28591 tablet_replica.cc:330] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: stopping tablet replica
I20260619 02:56:55.312520 28589 raft_consensus.cc:2258] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:55.312855 28591 raft_consensus.cc:2229] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:56:55.313416 28591 raft_consensus.cc:2258] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:55.315084 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28161
I20260619 02:56:55.315975 28591 ts_tablet_manager.cc:1905] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:55.320005 28589 ts_tablet_manager.cc:1905] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:55.325057 28591 ts_tablet_manager.cc:1918] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:55.325299 28591 log.cc:1196] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/cf2828d50af341b29111cbc514825aea
I20260619 02:56:55.325553 28590 tablet_replica.cc:330] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: stopping tablet replica
I20260619 02:56:55.325930 28591 ts_tablet_manager.cc:1939] T cf2828d50af341b29111cbc514825aea P 78aa2a60e563479a89bb03f635e043e2: Deleting consensus metadata
I20260619 02:56:55.325944 28590 raft_consensus.cc:2229] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:56:55.326381 28590 raft_consensus.cc:2258] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:56:55.327705 28087 catalog_manager.cc:4841] TS 78aa2a60e563479a89bb03f635e043e2 (127.22.230.2:44571): tablet cf2828d50af341b29111cbc514825aea (table test-table [id=62711fb31f634ea58f4cd01a06a65cc7]) successfully deleted
I20260619 02:56:55.328356 28589 ts_tablet_manager.cc:1918] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:55.328503 28589 log.cc:1196] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/cf2828d50af341b29111cbc514825aea
I20260619 02:56:55.329051 28589 ts_tablet_manager.cc:1939] T cf2828d50af341b29111cbc514825aea P 3bcd806f194f4fc1b7cf6c6bdd4db435: Deleting consensus metadata
I20260619 02:56:55.329214 28590 ts_tablet_manager.cc:1905] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:56:55.329869 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.316057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.22.230.1:40863)
0619 02:56:55.316206 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.316213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.327224 (+ 11011us) server_negotiation.cc:407] Connection header received
0619 02:56:55.327391 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.327397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.327487 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.327583 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.329109 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.329115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.329119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.329137 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.329174 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.329193 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.329287 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.329542 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.329595 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:56:55.330825 28283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:55.330791 28086 catalog_manager.cc:4841] TS 3bcd806f194f4fc1b7cf6c6bdd4db435 (127.22.230.1:40863): tablet cf2828d50af341b29111cbc514825aea (table test-table [id=62711fb31f634ea58f4cd01a06a65cc7]) successfully deleted
I20260619 02:56:55.335358 28590 ts_tablet_manager.cc:1918] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:56:55.335510 28590 log.cc:1196] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/cf2828d50af341b29111cbc514825aea
I20260619 02:56:55.335949 28590 ts_tablet_manager.cc:1939] T cf2828d50af341b29111cbc514825aea P 31c47467a6e0446d97d8b314ea054025: Deleting consensus metadata
I20260619 02:56:55.337369 28089 catalog_manager.cc:4841] TS 31c47467a6e0446d97d8b314ea054025 (127.22.230.3:39103): tablet cf2828d50af341b29111cbc514825aea (table test-table [id=62711fb31f634ea58f4cd01a06a65cc7]) successfully deleted
I20260619 02:56:55.424739 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28161
I20260619 02:56:55.441515 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28291
I20260619 02:56:55.444315 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.442395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.22.230.2:44571)
0619 02:56:55.442654 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.442659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.442674 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:56:55.442738 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.442741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.442786 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.442886 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.443727 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.443731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.443734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.443746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.443773 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.443783 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.443908 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.444005 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.444059 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:56:55.444958 28413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:55.547189 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28291
I20260619 02:56:55.565950 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28421
I20260619 02:56:55.568737 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:55.566657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.22.230.3:39103)
0619 02:56:55.566838 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:55.566845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:55.566909 (+ 64us) server_negotiation.cc:407] Connection header received
0619 02:56:55.567152 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:55.567162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:55.567230 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:55.567334 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:55.568122 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:55.568127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:55.568132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:55.568149 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:55.568200 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:55.568211 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:55.568332 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:55.568407 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 02:56:55.568461 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:56:55.569293 28543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:55.652249 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28421
I20260619 02:56:55.667568 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28071
I20260619 02:56:55.668634 28132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:55.782482 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28071
2026-06-19T02:56:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2465 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 02:56:55.818385 23448 mini_oidc.cc:152] Starting JWKS server
I20260619 02:56:55.820983 23448 webserver.cc:460] Webserver started at https://127.0.0.1:33293/ using document root <none> and password file <none>
I20260619 02:56:55.821322 23448 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:56:55.821918 23448 webserver.cc:460] Webserver started at http://127.0.0.1:37925/ using document root <none> and password file <none>
2026-06-19T02:56:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:55Z Disabled control of system clock
I20260619 02:56:55.847190 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:41287
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:41287
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33293/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/testchainca.cert with env {}
W20260619 02:56:56.147903 28602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:56.148161 28602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:56.148335 28602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:56.155200 28602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:56.155313 28602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:56.155385 28602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:56.155478 28602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:56.155555 28602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:56:56.155674 28602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:56:56.155745 28602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33293/jwks.json
I20260619 02:56:56.164636 28602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:41287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:41287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33293/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:56.165776 28602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:56.167001 28602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:56.177750 28608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:56.177820 28607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:56.177815 28610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:56.178810 28602 server_base.cc:1034] running on GCE node
I20260619 02:56:56.179452 28602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:56.180673 28602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:56.181880 28602 hybrid_clock.cc:648] HybridClock initialized: now 1781837816181851 us; error 42 us; skew 500 ppm
I20260619 02:56:56.182181 28602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:56.184213 28602 webserver.cc:460] Webserver started at http://127.22.230.62:41481/ using document root <none> and password file <none>
I20260619 02:56:56.184836 28602 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:56.184960 28602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:56.185243 28602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:56.187949 28602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "4f6ef4f35b7243c29941703733e066fd"
format_stamp: "Formatted at 2026-06-19 02:56:56 on dist-test-slave-2gms"
server_key: "94a3ae09b0a511416ec50b8b821388d5"
server_key_iv: "fab9c2da50623be57c77b5ec380c1e83"
server_key_version: "encryptionkey@0"
I20260619 02:56:56.188687 28602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "4f6ef4f35b7243c29941703733e066fd"
format_stamp: "Formatted at 2026-06-19 02:56:56 on dist-test-slave-2gms"
server_key: "94a3ae09b0a511416ec50b8b821388d5"
server_key_iv: "fab9c2da50623be57c77b5ec380c1e83"
server_key_version: "encryptionkey@0"
I20260619 02:56:56.193277 28602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:56:56.196352 28616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:56.197525 28602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:56:56.197698 28602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "4f6ef4f35b7243c29941703733e066fd"
format_stamp: "Formatted at 2026-06-19 02:56:56 on dist-test-slave-2gms"
server_key: "94a3ae09b0a511416ec50b8b821388d5"
server_key_iv: "fab9c2da50623be57c77b5ec380c1e83"
server_key_version: "encryptionkey@0"
I20260619 02:56:56.197883 28602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:56.213304 28602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:56.213953 28602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:56.214208 28602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:56.230980 28602 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:41287
I20260619 02:56:56.230994 28667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:41287 every 8 connection(s)
I20260619 02:56:56.232143 28602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:56.235486 28668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:56.241762 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28602
I20260619 02:56:56.241981 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:56.242761 23448 external_mini_cluster.cc:1428] Setting key be8984239a8f3b6b44ef21a1a839a2ff
I20260619 02:56:56.245682 28668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd: Bootstrap starting.
I20260619 02:56:56.248621 28668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:56.249588 28668 log.cc:824] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:56.252362 28668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd: No bootstrap required, opened a new log
I20260619 02:56:56.255792 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:56.244507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.22.230.62:41287)
0619 02:56:56.245376 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:56.245386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:56.245423 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:56:56.247218 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:56.247240 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:56.247638 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:56.247978 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:56.248473 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.249801 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.251722 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.252332 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.252499 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:56.253645 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:56.253651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:56.253671 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:56.254161 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:56.254239 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:56.254257 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:56.254341 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:56.254604 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 02:56:56.254931 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":294,"threads_started":1}
I20260619 02:56:56.259439 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:56:56.261384 28668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6ef4f35b7243c29941703733e066fd" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41287 } }
I20260619 02:56:56.261786 28668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:56.261924 28668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6ef4f35b7243c29941703733e066fd, State: Initialized, Role: FOLLOWER
I20260619 02:56:56.262485 28668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6ef4f35b7243c29941703733e066fd" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41287 } }
I20260619 02:56:56.262776 28668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:56.262907 28668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:56.263111 28668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:56.265074 28668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6ef4f35b7243c29941703733e066fd" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41287 } }
I20260619 02:56:56.265523 28668 leader_election.cc:302] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f6ef4f35b7243c29941703733e066fd; no voters:
I20260619 02:56:56.266065 28668 leader_election.cc:288] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:56.266198 28673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:56.266933 28673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [term 1 LEADER]: Becoming Leader. State: Replica: 4f6ef4f35b7243c29941703733e066fd, State: Running, Role: LEADER
I20260619 02:56:56.267467 28673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6ef4f35b7243c29941703733e066fd" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41287 } }
I20260619 02:56:56.268407 28668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:56.270213 28674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f6ef4f35b7243c29941703733e066fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6ef4f35b7243c29941703733e066fd" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41287 } } }
I20260619 02:56:56.270684 28674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:56.271255 28675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f6ef4f35b7243c29941703733e066fd. Latest consensus state: current_term: 1 leader_uuid: "4f6ef4f35b7243c29941703733e066fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6ef4f35b7243c29941703733e066fd" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41287 } } }
I20260619 02:56:56.271624 28675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:56.273218 28682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:56.276971 28682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:56.284713 28682 catalog_manager.cc:1347] Generated new cluster ID: fc5738cf3d9346f9b01e5bdccae8d359
I20260619 02:56:56.284811 28682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:56.303715 28682 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:56.305416 28682 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:56.313712 28682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f6ef4f35b7243c29941703733e066fd: Generated new TSK 0
I20260619 02:56:56.314698 28682 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:56:56.364298 28698 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 02:56:56.364470 28698 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:56:56.365475 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:56.339707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.22.230.62:41287)
0619 02:56:56.339943 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:56.339950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:56.340305 (+ 355us) server_negotiation.cc:407] Connection header received
0619 02:56:56.340585 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:56.340588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:56.340649 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:56.340754 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:56:56.341333 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.342020 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.343980 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.344415 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.344527 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:56.345065 (+ 538us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:56:56.364893 (+ 19828us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:56:56.365210 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 02:56:56.365593 28671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 02:56:56.366776 28697 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:56:56.339589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:41287 (local address 127.0.0.1:36678)
0619 02:56:56.340204 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:56.340225 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:56.340387 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:56.340806 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:56.340812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:56.340825 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:56:56.341196 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:56.341206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.342145 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.342150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:56.343859 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:56.343872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.344581 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.344588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:56.344778 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:56.344843 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:56:56.365139 (+ 20296us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 02:56:56.365360 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.230.62:41287: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":322,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.230.62:41287: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 02:56:56.368182 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28602
I20260619 02:56:56.369256 28663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:56.458335 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28602
2026-06-19T02:56:56Z chronyd exiting
I20260619 02:56:56.508139 23448 test_util.cc:175] -----------------------------------------------
I20260619 02:56:56.508270 23448 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781837774831470-23448-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (693 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 02:56:56.511677 23448 mini_oidc.cc:152] Starting JWKS server
I20260619 02:56:56.513358 23448 webserver.cc:460] Webserver started at https://127.0.0.1:44267/ using document root <none> and password file <none>
I20260619 02:56:56.513509 23448 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:56:56.514050 23448 webserver.cc:460] Webserver started at http://127.0.0.1:34495/ using document root <none> and password file <none>
2026-06-19T02:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:56Z Disabled control of system clock
I20260619 02:56:56.538889 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45941
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:45941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44267/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/testchainca.cert with env {}
W20260619 02:56:56.843370 28705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:56.843621 28705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:56.843696 28705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:56.850780 28705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:56.850871 28705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:56.850907 28705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:56.850962 28705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:56.851009 28705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:56:56.851070 28705 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:56:56.851109 28705 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44267/jwks.json
I20260619 02:56:56.860544 28705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:45941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44267/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:56.861641 28705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:56.862915 28705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:56.874076 28711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:56.874118 28710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:56.874151 28713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:56.874825 28705 server_base.cc:1034] running on GCE node
I20260619 02:56:56.875579 28705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:56.876820 28705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:56.878038 28705 hybrid_clock.cc:648] HybridClock initialized: now 1781837816877997 us; error 51 us; skew 500 ppm
I20260619 02:56:56.878358 28705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:56.880499 28705 webserver.cc:460] Webserver started at http://127.22.230.62:42323/ using document root <none> and password file <none>
I20260619 02:56:56.881136 28705 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:56.881268 28705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:56.881562 28705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:56.884244 28705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "cb726c83285244cda6c1a709d61ebc02"
format_stamp: "Formatted at 2026-06-19 02:56:56 on dist-test-slave-2gms"
server_key: "3ad31774cd010570539d7c04efe5ff74"
server_key_iv: "02a2be50b1d540b7154800f76c9233f5"
server_key_version: "encryptionkey@0"
I20260619 02:56:56.885020 28705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "cb726c83285244cda6c1a709d61ebc02"
format_stamp: "Formatted at 2026-06-19 02:56:56 on dist-test-slave-2gms"
server_key: "3ad31774cd010570539d7c04efe5ff74"
server_key_iv: "02a2be50b1d540b7154800f76c9233f5"
server_key_version: "encryptionkey@0"
I20260619 02:56:56.889668 28705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:56:56.892817 28719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:56.893960 28705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:56:56.894138 28705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "cb726c83285244cda6c1a709d61ebc02"
format_stamp: "Formatted at 2026-06-19 02:56:56 on dist-test-slave-2gms"
server_key: "3ad31774cd010570539d7c04efe5ff74"
server_key_iv: "02a2be50b1d540b7154800f76c9233f5"
server_key_version: "encryptionkey@0"
I20260619 02:56:56.894338 28705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:56.909039 28705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:56.910526 28705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:56.910859 28705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:56.928215 28705 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:45941
I20260619 02:56:56.928233 28770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:45941 every 8 connection(s)
I20260619 02:56:56.929484 28705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:56.933199 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:56.933223 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28705
I20260619 02:56:56.933399 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:56.934237 23448 external_mini_cluster.cc:1428] Setting key 10f93d5ee72b2f5a79b7562ec5cfd55e
I20260619 02:56:56.944991 28771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02: Bootstrap starting.
I20260619 02:56:56.947204 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:56.936065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.22.230.62:45941)
0619 02:56:56.937003 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:56.937012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:56.937048 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:56:56.939245 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:56.939266 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:56.939705 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:56.940090 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:56.940582 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.941728 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.943008 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:56.943623 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:56.943828 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:56.944976 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:56.944986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:56.944999 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:56.945539 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:56.945607 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:56.945626 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:56.945743 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:56.945977 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 02:56:56.946249 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":351,"threads_started":1}
I20260619 02:56:56.948195 28771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:56.949503 28771 log.cc:824] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:56.950954 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:56:56.952266 28771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02: No bootstrap required, opened a new log
I20260619 02:56:56.959609 28771 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb726c83285244cda6c1a709d61ebc02" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45941 } }
I20260619 02:56:56.960018 28771 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:56.960130 28771 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb726c83285244cda6c1a709d61ebc02, State: Initialized, Role: FOLLOWER
I20260619 02:56:56.960665 28771 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb726c83285244cda6c1a709d61ebc02" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45941 } }
I20260619 02:56:56.960887 28771 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:56.961040 28771 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:56.961203 28771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:56.963290 28771 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb726c83285244cda6c1a709d61ebc02" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45941 } }
I20260619 02:56:56.963752 28771 leader_election.cc:302] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb726c83285244cda6c1a709d61ebc02; no voters:
I20260619 02:56:56.964293 28771 leader_election.cc:288] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:56.964426 28776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:56.966071 28776 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [term 1 LEADER]: Becoming Leader. State: Replica: cb726c83285244cda6c1a709d61ebc02, State: Running, Role: LEADER
I20260619 02:56:56.966318 28771 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:56.966688 28776 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb726c83285244cda6c1a709d61ebc02" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45941 } }
I20260619 02:56:56.969410 28778 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb726c83285244cda6c1a709d61ebc02. Latest consensus state: current_term: 1 leader_uuid: "cb726c83285244cda6c1a709d61ebc02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb726c83285244cda6c1a709d61ebc02" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45941 } } }
I20260619 02:56:56.969861 28778 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:56.969452 28777 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb726c83285244cda6c1a709d61ebc02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb726c83285244cda6c1a709d61ebc02" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45941 } } }
I20260619 02:56:56.970973 28777 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:56.975128 28786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:56.979251 28786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:56.987123 28786 catalog_manager.cc:1347] Generated new cluster ID: 3ffca0537ad348d19c1a56e09516f2f9
I20260619 02:56:56.987215 28786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:57.005775 28786 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:57.007244 28786 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:57.021303 28786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb726c83285244cda6c1a709d61ebc02: Generated new TSK 0
I20260619 02:56:57.022058 28786 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 02:56:57.059131 28801 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 02:56:57.059247 28801 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 02:56:57.060374 28800 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:56:57.040277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45941 (local address 127.0.0.1:35280)
0619 02:56:57.040898 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:57.040913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:57.041020 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:57.041425 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:57.041428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:57.041437 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:56:57.041685 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:57.041692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.043117 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.043120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:57.044166 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:57.044175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.045147 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.045151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:57.045282 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:57.045311 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:56:57.059931 (+ 14620us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:56:57.060096 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.230.62:45941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":345,"threads_started":1}
I20260619 02:56:57.060391 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:57.040285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.22.230.62:45941)
0619 02:56:57.040459 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:57.040467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:57.040991 (+ 524us) server_negotiation.cc:407] Connection header received
0619 02:56:57.041169 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:57.041175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:57.041263 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:57.041413 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:56:57.041826 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.042997 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.044315 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.044991 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.045124 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:57.045561 (+ 437us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:56:57.059747 (+ 14186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 02:56:57.060090 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 02:56:57.060508 28774 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44267/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 02:56:57.061820 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28705
I20260619 02:56:57.062889 28766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:57.153744 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28705
2026-06-19T02:56:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 02:56:57.208362 23448 mini_oidc.cc:152] Starting JWKS server
I20260619 02:56:57.210260 23448 webserver.cc:460] Webserver started at https://127.0.0.1:45737/ using document root <none> and password file <none>
I20260619 02:56:57.210471 23448 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 02:56:57.211037 23448 webserver.cc:460] Webserver started at http://127.0.0.1:37641/ using document root <none> and password file <none>
2026-06-19T02:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:57Z Disabled control of system clock
I20260619 02:56:57.236483 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38977
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45737/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:56:57.537525 28808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:57.537765 28808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:57.537843 28808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:57.544632 28808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:57.544723 28808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:57.544760 28808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:57.544806 28808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:57.544849 28808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:56:57.544901 28808 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:56:57.544940 28808 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45737/jwks.json
I20260619 02:56:57.553589 28808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45737/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:57.554689 28808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:57.556377 28808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:57.566804 28813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:57.567528 28814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:57.568344 28816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:57.569116 28808 server_base.cc:1034] running on GCE node
I20260619 02:56:57.569792 28808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:57.571097 28808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:57.572289 28808 hybrid_clock.cc:648] HybridClock initialized: now 1781837817572243 us; error 35 us; skew 500 ppm
I20260619 02:56:57.572556 28808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:57.578737 28808 webserver.cc:460] Webserver started at http://127.22.230.62:40463/ using document root <none> and password file <none>
I20260619 02:56:57.579319 28808 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:57.579399 28808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:57.579684 28808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:57.582165 28808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "089e4aaede9c401aa253eb9fc7a2eedf"
format_stamp: "Formatted at 2026-06-19 02:56:57 on dist-test-slave-2gms"
server_key: "1f04b2d291584728ff73ab8ac30f56e7"
server_key_iv: "cb02e042673a32adfec1d5532fc4b32c"
server_key_version: "encryptionkey@0"
I20260619 02:56:57.582873 28808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "089e4aaede9c401aa253eb9fc7a2eedf"
format_stamp: "Formatted at 2026-06-19 02:56:57 on dist-test-slave-2gms"
server_key: "1f04b2d291584728ff73ab8ac30f56e7"
server_key_iv: "cb02e042673a32adfec1d5532fc4b32c"
server_key_version: "encryptionkey@0"
I20260619 02:56:57.587494 28808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:56:57.590639 28822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:57.591851 28808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:56:57.592064 28808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "089e4aaede9c401aa253eb9fc7a2eedf"
format_stamp: "Formatted at 2026-06-19 02:56:57 on dist-test-slave-2gms"
server_key: "1f04b2d291584728ff73ab8ac30f56e7"
server_key_iv: "cb02e042673a32adfec1d5532fc4b32c"
server_key_version: "encryptionkey@0"
I20260619 02:56:57.592257 28808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:57.610078 28808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:57.610774 28808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:57.611060 28808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:57.628011 28808 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:38977
I20260619 02:56:57.628019 28873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:38977 every 8 connection(s)
I20260619 02:56:57.629192 28808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:57.631371 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28808
I20260619 02:56:57.631579 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:57.632320 23448 external_mini_cluster.cc:1428] Setting key 352e98f8bb726d02d55981a0e9257ccd
I20260619 02:56:57.633020 28874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:57.644393 28874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf: Bootstrap starting.
I20260619 02:56:57.645176 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:57.634062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.22.230.62:38977)
0619 02:56:57.634994 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:57.635005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:57.635044 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:56:57.637251 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:57.637270 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:57.637672 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:57.638038 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:57.638652 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.639878 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.641093 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.641710 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.641910 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:57.643122 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:57.643129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:57.643141 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:57.643618 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:57.643685 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:57.643702 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:57.643816 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:57.644029 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 02:56:57.644285 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":319,"threads_started":1}
I20260619 02:56:57.647697 28874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:57.648695 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:56:57.648813 28874 log.cc:824] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:57.651396 28874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf: No bootstrap required, opened a new log
I20260619 02:56:57.658203 28874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38977 } }
I20260619 02:56:57.658504 28874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:57.658649 28874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089e4aaede9c401aa253eb9fc7a2eedf, State: Initialized, Role: FOLLOWER
I20260619 02:56:57.659125 28874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38977 } }
I20260619 02:56:57.659368 28874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:57.659502 28874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:57.659658 28874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:57.661759 28874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38977 } }
I20260619 02:56:57.662207 28874 leader_election.cc:302] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 089e4aaede9c401aa253eb9fc7a2eedf; no voters:
I20260619 02:56:57.662779 28874 leader_election.cc:288] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:57.662895 28879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:57.663635 28879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [term 1 LEADER]: Becoming Leader. State: Replica: 089e4aaede9c401aa253eb9fc7a2eedf, State: Running, Role: LEADER
I20260619 02:56:57.664181 28879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38977 } }
I20260619 02:56:57.664731 28874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:57.666802 28881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 089e4aaede9c401aa253eb9fc7a2eedf. Latest consensus state: current_term: 1 leader_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38977 } } }
I20260619 02:56:57.667263 28881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:57.667825 28880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "089e4aaede9c401aa253eb9fc7a2eedf" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38977 } } }
I20260619 02:56:57.668164 28880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:57.668771 28886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:57.673291 28886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:57.680576 28886 catalog_manager.cc:1347] Generated new cluster ID: ec63dc766ba24efcb3f7be3fd921388d
I20260619 02:56:57.680675 28886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:57.699440 28886 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:57.700867 28886 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:57.710254 28886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 089e4aaede9c401aa253eb9fc7a2eedf: Generated new TSK 0
I20260619 02:56:57.711042 28886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:56:57.740453 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:57.732463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.22.230.62:38977)
0619 02:56:57.732645 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:57.732651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:57.733045 (+ 394us) server_negotiation.cc:407] Connection header received
0619 02:56:57.733234 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:57.733240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:57.733320 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:57.733463 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:56:57.733927 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.735058 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.736385 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.737038 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.737185 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:57.737648 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:56:57.739852 (+ 2204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45737/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:56:57.740150 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52220: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45737/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 02:56:57.740571 28877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52220: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45737/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 02:56:57.742183 28903 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:56:57.732336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38977 (local address 127.0.0.1:52220)
0619 02:56:57.732940 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:57.732955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:57.733067 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:57.733512 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:57.733516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:57.733526 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:56:57.733790 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:57.733798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.735216 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.735220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:57.736254 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:57.736265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:57.737232 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:57.737237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:57.737356 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:57.737386 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:56:57.740079 (+ 2693us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45737/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 02:56:57.741785 (+ 1706us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.230.62:38977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45737/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":1402,"thread_start_us":335,"threads_started":1}
I20260619 02:56:57.743176 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28808
I20260619 02:56:57.744241 28869 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 02:56:57.752398 28904 webserver.cc:533] Webserver: sslize failed:
E20260619 02:56:57.752509 28904 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 02:56:57.832188 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28808
2026-06-19T02:56:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (679 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T02:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:56:57Z Disabled control of system clock
I20260619 02:56:57.913194 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:40315
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:40315
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/testchainca.cert with env {}
W20260619 02:56:58.217626 28909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:56:58.217911 28909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:56:58.218068 28909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:56:58.225176 28909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:56:58.225268 28909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:56:58.225350 28909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:56:58.225452 28909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:56:58.225534 28909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 02:56:58.225646 28909 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 02:56:58.225723 28909 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 02:56:58.234973 28909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:40315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:56:58.236177 28909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:56:58.237380 28909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:56:58.248453 28914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:58.248486 28917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:56:58.248512 28915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:56:58.249570 28909 server_base.cc:1034] running on GCE node
I20260619 02:56:58.250252 28909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:56:58.251626 28909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:56:58.252846 28909 hybrid_clock.cc:648] HybridClock initialized: now 1781837818252814 us; error 50 us; skew 500 ppm
I20260619 02:56:58.253185 28909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:56:58.255499 28909 webserver.cc:460] Webserver started at http://127.22.230.62:39167/ using document root <none> and password file <none>
I20260619 02:56:58.256140 28909 fs_manager.cc:359] Metadata directory not provided
I20260619 02:56:58.256270 28909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:56:58.256564 28909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:56:58.259280 28909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "e0da8ee336304b6b91baa6710e8bbe47"
format_stamp: "Formatted at 2026-06-19 02:56:58 on dist-test-slave-2gms"
server_key: "fabb29b036ff62c63378422a35e090d7"
server_key_iv: "2e5767dc65480d19121dd5a042f478fb"
server_key_version: "encryptionkey@0"
I20260619 02:56:58.260051 28909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "e0da8ee336304b6b91baa6710e8bbe47"
format_stamp: "Formatted at 2026-06-19 02:56:58 on dist-test-slave-2gms"
server_key: "fabb29b036ff62c63378422a35e090d7"
server_key_iv: "2e5767dc65480d19121dd5a042f478fb"
server_key_version: "encryptionkey@0"
I20260619 02:56:58.264487 28909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:56:58.267568 28923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:56:58.268756 28909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:56:58.268925 28909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "e0da8ee336304b6b91baa6710e8bbe47"
format_stamp: "Formatted at 2026-06-19 02:56:58 on dist-test-slave-2gms"
server_key: "fabb29b036ff62c63378422a35e090d7"
server_key_iv: "2e5767dc65480d19121dd5a042f478fb"
server_key_version: "encryptionkey@0"
I20260619 02:56:58.269127 28909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:56:58.281198 28909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:56:58.281862 28909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:56:58.282136 28909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:56:58.298700 28909 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:40315
I20260619 02:56:58.298676 28974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:40315 every 8 connection(s)
I20260619 02:56:58.299889 28909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:56:58.300623 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 28909
I20260619 02:56:58.300833 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:56:58.301646 23448 external_mini_cluster.cc:1428] Setting key d091039a1cd548ec195268001fcabafd
I20260619 02:56:58.304260 28975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:56:58.314199 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:58.303416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.22.230.62:40315)
0619 02:56:58.304330 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:58.304343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:58.304389 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:56:58.305886 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:58.305904 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:58.306292 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:58.306653 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:56:58.307179 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:58.308454 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:58.310239 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:58.310900 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:58.311014 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:58.312145 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:56:58.312151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:56:58.312162 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:56:58.312644 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:56:58.312718 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:56:58.312738 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:56:58.312820 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 02:56:58.313035 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 02:56:58.313340 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":341,"threads_started":1}
I20260619 02:56:58.315543 28975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47: Bootstrap starting.
I20260619 02:56:58.317835 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:56:58.319038 28975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47: Neither blocks nor log segments found. Creating new log.
I20260619 02:56:58.320118 28975 log.cc:824] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47: Log is configured to *not* fsync() on all Append() calls
I20260619 02:56:58.322927 28975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47: No bootstrap required, opened a new log
I20260619 02:56:58.329887 28975 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0da8ee336304b6b91baa6710e8bbe47" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40315 } }
I20260619 02:56:58.330168 28975 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:56:58.330241 28975 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0da8ee336304b6b91baa6710e8bbe47, State: Initialized, Role: FOLLOWER
I20260619 02:56:58.330732 28975 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0da8ee336304b6b91baa6710e8bbe47" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40315 } }
I20260619 02:56:58.330922 28975 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:56:58.331010 28975 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:56:58.331130 28975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:56:58.333240 28975 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0da8ee336304b6b91baa6710e8bbe47" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40315 } }
I20260619 02:56:58.333643 28975 leader_election.cc:302] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0da8ee336304b6b91baa6710e8bbe47; no voters:
I20260619 02:56:58.334095 28975 leader_election.cc:288] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:56:58.334237 28980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:56:58.334911 28980 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [term 1 LEADER]: Becoming Leader. State: Replica: e0da8ee336304b6b91baa6710e8bbe47, State: Running, Role: LEADER
I20260619 02:56:58.335412 28980 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0da8ee336304b6b91baa6710e8bbe47" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40315 } }
I20260619 02:56:58.336037 28975 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:56:58.338106 28982 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0da8ee336304b6b91baa6710e8bbe47. Latest consensus state: current_term: 1 leader_uuid: "e0da8ee336304b6b91baa6710e8bbe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0da8ee336304b6b91baa6710e8bbe47" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40315 } } }
I20260619 02:56:58.338511 28982 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:58.338311 28981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0da8ee336304b6b91baa6710e8bbe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0da8ee336304b6b91baa6710e8bbe47" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 40315 } } }
I20260619 02:56:58.338654 28981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47 [sys.catalog]: This master's current role is: LEADER
I20260619 02:56:58.340170 28985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:56:58.344611 28985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:56:58.351702 28985 catalog_manager.cc:1347] Generated new cluster ID: 3818cc45b41040589542f53d4d529119
I20260619 02:56:58.351792 28985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:56:58.370216 28985 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:56:58.371960 28985 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:56:58.386904 28985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0da8ee336304b6b91baa6710e8bbe47: Generated new TSK 0
I20260619 02:56:58.387816 28985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:56:58.425824 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:56:58.401500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.22.230.62:40315)
0619 02:56:58.401641 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 02:56:58.401645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:56:58.401986 (+ 341us) server_negotiation.cc:407] Connection header received
0619 02:56:58.402223 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:56:58.402227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:56:58.402292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:56:58.402430 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0619 02:56:58.402951 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:58.403642 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:58.405592 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:56:58.406033 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:56:58.406120 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:58.406697 (+ 577us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 02:56:58.425331 (+ 18634us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:56:58.425601 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37420: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 02:56:58.425972 28978 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37420: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 02:56:58.426254 29004 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:56:58.401325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:40315 (local address 127.0.0.1:37420)
0619 02:56:58.401884 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 02:56:58.401902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:56:58.402037 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:56:58.402473 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:56:58.402478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:56:58.402490 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 02:56:58.402804 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:58.402817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:58.403774 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:58.403780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:58.405456 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:56:58.405471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:56:58.406188 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:56:58.406194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:56:58.406381 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:56:58.406423 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 02:56:58.425598 (+ 19175us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 02:56:58.425860 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.230.62:40315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":323,"threads_started":1}
I20260619 02:56:58.427398 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 28909
I20260619 02:56:58.428498 28970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:56:58.516463 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 28909
2026-06-19T02:56:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (662 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (386 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (387 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:56:59 dist-test-slave-2gms krb5kdc[29019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:56:59 dist-test-slave-2gms krb5kdc[29019](info): set up 2 sockets
Jun 19 02:56:59 dist-test-slave-2gms krb5kdc[29019](info): commencing operation
krb5kdc: starting...
W20260619 02:57:00.363243 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:00 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:57:00.545650 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33927
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:33927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:00.854732 29032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:00.855005 29032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:00.855096 29032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:00.861953 29032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:00.862036 29032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:00.862090 29032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:00.862136 29032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:00.862177 29032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:00.870995 29032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:33927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:00.872071 29032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:00.873194 29032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:00.884037 29037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:00.884076 29040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:00.884094 29038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:00.884963 29032 server_base.cc:1034] running on GCE node
I20260619 02:57:00.885711 29032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:00.886976 29032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:00.888211 29032 hybrid_clock.cc:648] HybridClock initialized: now 1781837820888145 us; error 49 us; skew 500 ppm
I20260619 02:57:00.888513 29032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:00 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:00.891345 29032 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:57:00.892697 29032 webserver.cc:460] Webserver started at http://127.22.230.62:41195/ using document root <none> and password file <none>
I20260619 02:57:00.893311 29032 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:00.893445 29032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:00.893740 29032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:00.896505 29032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "6b436cf5a17642b590ce67451c74a58d"
format_stamp: "Formatted at 2026-06-19 02:57:00 on dist-test-slave-2gms"
server_key: "e4fe219cb225dcda8113366aac99b69a"
server_key_iv: "421c9b493322140b0c22f59bcea29d66"
server_key_version: "encryptionkey@0"
I20260619 02:57:00.897219 29032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "6b436cf5a17642b590ce67451c74a58d"
format_stamp: "Formatted at 2026-06-19 02:57:00 on dist-test-slave-2gms"
server_key: "e4fe219cb225dcda8113366aac99b69a"
server_key_iv: "421c9b493322140b0c22f59bcea29d66"
server_key_version: "encryptionkey@0"
I20260619 02:57:00.901893 29032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:57:00.905009 29047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:00.906201 29032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:57:00.906388 29032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "6b436cf5a17642b590ce67451c74a58d"
format_stamp: "Formatted at 2026-06-19 02:57:00 on dist-test-slave-2gms"
server_key: "e4fe219cb225dcda8113366aac99b69a"
server_key_iv: "421c9b493322140b0c22f59bcea29d66"
server_key_version: "encryptionkey@0"
I20260619 02:57:00.906610 29032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:00.932729 29032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:00.936316 29032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:00.936576 29032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:00.954370 29032 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:33927
I20260619 02:57:00.954392 29098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:33927 every 8 connection(s)
I20260619 02:57:00.955665 29032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:00.959082 29099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:00.962476 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29032
I20260619 02:57:00.962700 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:00.963529 23448 external_mini_cluster.cc:1428] Setting key ced40bb6980ff6f0ab391c4086b39cb0
I20260619 02:57:00.970971 29099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d: Bootstrap starting.
I20260619 02:57:00.973670 29099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:00.974620 29099 log.cc:824] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:57:00 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:00.977598 29099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d: No bootstrap required, opened a new log
I20260619 02:57:00.984807 29102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:00.965485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.22.230.62:33927)
0619 02:57:00.966195 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:00.966224 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:00.966262 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:57:00.967885 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:00.967905 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:00.968273 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:00.968644 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:00.969912 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:00.971274 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:00.972511 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:00.973119 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:00.973298 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:00.977888 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:00.977896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:00.977909 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:00.977949 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:00.981051 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:00.981793 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:00.981798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:00.981804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:00.981954 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:00.982335 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:00.982337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:00.982339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:00.982874 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:00.983099 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:00.983494 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 02:57:00.983813 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":311,"threads_started":1}
I20260619 02:57:00.985870 29099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b436cf5a17642b590ce67451c74a58d" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33927 } }
I20260619 02:57:00.986196 29099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:00.986294 29099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b436cf5a17642b590ce67451c74a58d, State: Initialized, Role: FOLLOWER
I20260619 02:57:00.986912 29099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b436cf5a17642b590ce67451c74a58d" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33927 } }
I20260619 02:57:00.987159 29099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:00.987303 29099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:00.987470 29099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:00.989399 29099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b436cf5a17642b590ce67451c74a58d" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33927 } }
I20260619 02:57:00.989845 29099 leader_election.cc:302] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b436cf5a17642b590ce67451c74a58d; no voters:
I20260619 02:57:00.990353 29099 leader_election.cc:288] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:00.990471 29104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:00.991199 29104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [term 1 LEADER]: Becoming Leader. State: Replica: 6b436cf5a17642b590ce67451c74a58d, State: Running, Role: LEADER
I20260619 02:57:00.991734 29104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b436cf5a17642b590ce67451c74a58d" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33927 } }
I20260619 02:57:00.992300 29099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:00.994235 29105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b436cf5a17642b590ce67451c74a58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b436cf5a17642b590ce67451c74a58d" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33927 } } }
I20260619 02:57:00.994544 29105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:00.994745 29106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b436cf5a17642b590ce67451c74a58d. Latest consensus state: current_term: 1 leader_uuid: "6b436cf5a17642b590ce67451c74a58d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b436cf5a17642b590ce67451c74a58d" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33927 } } }
I20260619 02:57:00.995142 29106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:00.996590 29109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:01.001690 29109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:01.009984 29109 catalog_manager.cc:1347] Generated new cluster ID: a146e0d60daf48f2b39d2cf43fddebf0
I20260619 02:57:01.010079 29109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:01.039960 29109 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:01.041172 29109 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:01.061184 29109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b436cf5a17642b590ce67451c74a58d: Generated new TSK 0
I20260619 02:57:01.062099 29109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:57:01.117972 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33927
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:01.432852 29125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:01.433148 29125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:01.433302 29125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:01.440459 29125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:01.440603 29125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:01.440768 29125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:01.450048 29125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:01.451340 29125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:01.452612 29125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:01.465539 29131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:01.465520 29133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:01.465772 29130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:01.466272 29125 server_base.cc:1034] running on GCE node
I20260619 02:57:01.466886 29125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:01.467782 29125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:01.468971 29125 hybrid_clock.cc:648] HybridClock initialized: now 1781837821468926 us; error 40 us; skew 500 ppm
I20260619 02:57:01.469260 29125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:01 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:01.472239 29125 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:57:01.473579 29125 webserver.cc:460] Webserver started at http://127.22.230.1:45711/ using document root <none> and password file <none>
I20260619 02:57:01.474205 29125 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:01.474330 29125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:01.474717 29125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:01.477473 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "451df00072d2424398ea2b3525f27b57"
format_stamp: "Formatted at 2026-06-19 02:57:01 on dist-test-slave-2gms"
server_key: "a49f6ed3262ab53f2a2edf4131b304f8"
server_key_iv: "ea200b68225cfa12109862cc5b616c61"
server_key_version: "encryptionkey@0"
I20260619 02:57:01.478276 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "451df00072d2424398ea2b3525f27b57"
format_stamp: "Formatted at 2026-06-19 02:57:01 on dist-test-slave-2gms"
server_key: "a49f6ed3262ab53f2a2edf4131b304f8"
server_key_iv: "ea200b68225cfa12109862cc5b616c61"
server_key_version: "encryptionkey@0"
I20260619 02:57:01.483327 29125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 02:57:01.487236 29140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:01.488444 29125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:57:01.488612 29125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "451df00072d2424398ea2b3525f27b57"
format_stamp: "Formatted at 2026-06-19 02:57:01 on dist-test-slave-2gms"
server_key: "a49f6ed3262ab53f2a2edf4131b304f8"
server_key_iv: "ea200b68225cfa12109862cc5b616c61"
server_key_version: "encryptionkey@0"
I20260619 02:57:01.488806 29125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:01.507618 29125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:01.511453 29125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:01.511730 29125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:01.512583 29125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:01.513826 29125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:01.513927 29125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:01.514079 29125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:01.514147 29125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:01.545426 29125 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:45323
I20260619 02:57:01.545478 29252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:45323 every 8 connection(s)
I20260619 02:57:01.546701 29125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:01.547102 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29125
I20260619 02:57:01.547354 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:01.548223 23448 external_mini_cluster.cc:1428] Setting key 8eb544f90c009f150004f56b1b992ed2
Jun 19 02:57:01 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:01.568059 29255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:01.549384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:41027 (local address 127.22.230.62:33927)
0619 02:57:01.549829 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:01.549837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:01.550951 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 02:57:01.552649 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:01.552654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:01.552749 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:01.552860 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:01.554301 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:01.555078 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:01.557188 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:01.557947 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:01.558096 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:01.561374 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:01.561382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:01.561390 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:01.561444 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:01.564981 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:01.565649 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:01.565657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:01.565664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:01.565815 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:01.566213 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:01.566221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:01.566227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:01.566646 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:01.566804 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:01.567397 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 02:57:01.567656 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260619 02:57:01.568682 29256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:01.550002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33927 (local address 127.22.230.1:41027)
0619 02:57:01.550801 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:01.550843 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:01.552409 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:01.553086 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:01.553095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:01.553595 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:01.554139 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:01.554157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:01.555218 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:01.555225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:01.557005 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:01.557022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:01.558155 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:01.558161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:01.558366 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:01.559091 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:01.559115 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:01.561124 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:01.565107 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:01.565113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:01.565125 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:01.565499 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:01.565931 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:01.565934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:01.565936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:01.566094 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:01.566758 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:01.566763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:01.567059 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 02:57:01.567274 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 02:57:01.567617 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:57:01.570225 29253 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33927
I20260619 02:57:01.570627 29253 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:01.571605 29253 heartbeater.cc:511] Master 127.22.230.62:33927 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:57:01.573918 29064 ts_manager.cc:194] Registered new tserver with Master: 451df00072d2424398ea2b3525f27b57 (127.22.230.1:45323)
I20260619 02:57:01.575325 29064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:41027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:57:01.596961 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33927
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:01.908712 29259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:01.909003 29259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:01.909104 29259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:01.917263 29259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:01.917375 29259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:01.917497 29259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:01.927407 29259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:01.928642 29259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:01.929857 29259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:01.942863 29264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:01.942800 29267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:01.942832 29265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:01.943795 29259 server_base.cc:1034] running on GCE node
I20260619 02:57:01.944378 29259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:01.945281 29259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:01.946498 29259 hybrid_clock.cc:648] HybridClock initialized: now 1781837821946460 us; error 50 us; skew 500 ppm
I20260619 02:57:01.946888 29259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:01 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:01.949882 29259 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:57:01.951287 29259 webserver.cc:460] Webserver started at http://127.22.230.2:45585/ using document root <none> and password file <none>
I20260619 02:57:01.951953 29259 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:01.952100 29259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:01.952397 29259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:01.955372 29259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "63a980fb640d4dc89bc99b3f7dcd037f"
format_stamp: "Formatted at 2026-06-19 02:57:01 on dist-test-slave-2gms"
server_key: "342a0abfd758cd53ca01b65359d4348b"
server_key_iv: "153816cb41b74bb6dc8faa6ce4ebf88a"
server_key_version: "encryptionkey@0"
I20260619 02:57:01.956118 29259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "63a980fb640d4dc89bc99b3f7dcd037f"
format_stamp: "Formatted at 2026-06-19 02:57:01 on dist-test-slave-2gms"
server_key: "342a0abfd758cd53ca01b65359d4348b"
server_key_iv: "153816cb41b74bb6dc8faa6ce4ebf88a"
server_key_version: "encryptionkey@0"
I20260619 02:57:01.961186 29259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:57:01.964745 29274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:01.966147 29259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:01.966317 29259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "63a980fb640d4dc89bc99b3f7dcd037f"
format_stamp: "Formatted at 2026-06-19 02:57:01 on dist-test-slave-2gms"
server_key: "342a0abfd758cd53ca01b65359d4348b"
server_key_iv: "153816cb41b74bb6dc8faa6ce4ebf88a"
server_key_version: "encryptionkey@0"
I20260619 02:57:01.966523 29259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:01.989941 29259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:01.994438 29259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:01.994762 29259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:01.995739 29259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:01.997031 29259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:01.997099 29259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:01.997227 29259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:01.997310 29259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:02.028982 29259 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:33483
I20260619 02:57:02.029093 29386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:33483 every 8 connection(s)
I20260619 02:57:02.030259 29259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:02.036494 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29259
I20260619 02:57:02.036686 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:02.037092 23448 external_mini_cluster.cc:1428] Setting key 1e002095fd72e779e02b9c7973fe1ea1
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:02.050882 29255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.032821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:47629 (local address 127.22.230.62:33927)
0619 02:57:02.033025 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.033030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.034146 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 02:57:02.035927 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.035931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.035995 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.036084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.037754 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.038771 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.040295 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.040703 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.040839 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.044609 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.044615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.044619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.044648 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.047605 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.048568 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.048572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.048574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.048696 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.049177 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.049180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.049182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.049430 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.049533 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.050429 (+ 896us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.050651 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 02:57:02.052559 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.033275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33927 (local address 127.22.230.2:47629)
0619 02:57:02.033979 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:02.034022 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:02.035677 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:02.036294 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:02.036301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:02.036943 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:02.037573 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:02.037592 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.038920 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.038930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:02.040148 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:02.040159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.040932 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.040939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:02.041193 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.041904 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:02.041927 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:02.044361 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:02.047746 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:02.047758 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:02.047778 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:02.048446 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:02.048803 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:02.048810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:02.048817 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:02.049067 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:02.049564 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:02.049575 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:02.050034 (+ 459us) client_negotiation.cc:769] Sending connection context
0619 02:57:02.050381 (+ 347us) client_negotiation.cc:240] Negotiation successful
0619 02:57:02.050995 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":282,"threads_started":1}
I20260619 02:57:02.054028 29387 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33927
I20260619 02:57:02.054365 29387 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:02.055301 29387 heartbeater.cc:511] Master 127.22.230.62:33927 requested a full tablet report, sending...
I20260619 02:57:02.056972 29064 ts_manager.cc:194] Registered new tserver with Master: 63a980fb640d4dc89bc99b3f7dcd037f (127.22.230.2:33483)
I20260619 02:57:02.058149 29064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:47629
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:57:02.090106 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33927
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:02.407665 29392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:02.407922 29392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:02.408033 29392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:02.414759 29392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:02.414873 29392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:02.414991 29392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:02.424202 29392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:02.425350 29392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:02.426530 29392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:02.438937 29400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:02.438977 29398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:02.439000 29397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:02.439774 29392 server_base.cc:1034] running on GCE node
I20260619 02:57:02.440304 29392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:02.441155 29392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:02.442337 29392 hybrid_clock.cc:648] HybridClock initialized: now 1781837822442285 us; error 49 us; skew 500 ppm
I20260619 02:57:02.442672 29392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.445920 29392 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:57:02.447386 29392 webserver.cc:460] Webserver started at http://127.22.230.3:40965/ using document root <none> and password file <none>
I20260619 02:57:02.448024 29392 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:02.448148 29392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:02.448443 29392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:02.451254 29392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "fc5e6f75ceed435fb86634488de8a5bb"
format_stamp: "Formatted at 2026-06-19 02:57:02 on dist-test-slave-2gms"
server_key: "2c779904063124dca8d67d52f4c04566"
server_key_iv: "8f6501c57b3826556b3518e1efb3f433"
server_key_version: "encryptionkey@0"
I20260619 02:57:02.452040 29392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "fc5e6f75ceed435fb86634488de8a5bb"
format_stamp: "Formatted at 2026-06-19 02:57:02 on dist-test-slave-2gms"
server_key: "2c779904063124dca8d67d52f4c04566"
server_key_iv: "8f6501c57b3826556b3518e1efb3f433"
server_key_version: "encryptionkey@0"
I20260619 02:57:02.457609 29392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:57:02.461118 29407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:02.462502 29392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:02.462726 29392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "fc5e6f75ceed435fb86634488de8a5bb"
format_stamp: "Formatted at 2026-06-19 02:57:02 on dist-test-slave-2gms"
server_key: "2c779904063124dca8d67d52f4c04566"
server_key_iv: "8f6501c57b3826556b3518e1efb3f433"
server_key_version: "encryptionkey@0"
I20260619 02:57:02.462929 29392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:02.480582 29392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:02.484367 29392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:02.484638 29392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:02.485523 29392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:02.486881 29392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:02.487001 29392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:02.487143 29392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:02.487219 29392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:02.519573 29392 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:46505
I20260619 02:57:02.519637 29519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:46505 every 8 connection(s)
I20260619 02:57:02.520864 29392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:02.531189 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29392
I20260619 02:57:02.531371 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:02.531840 23448 external_mini_cluster.cc:1428] Setting key 065db32e2c1b0ef682fc5778deea6f4c
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:02.540658 29255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.523217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:51761 (local address 127.22.230.62:33927)
0619 02:57:02.523421 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.523426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.524535 (+ 1109us) server_negotiation.cc:407] Connection header received
0619 02:57:02.526314 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.526320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.526408 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.526526 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.528003 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.528747 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.530277 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.530724 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.530854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.535079 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.535084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.535087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.535116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.537716 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.538428 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.538433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.538437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.538556 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.539025 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.539032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.539036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.539300 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.539456 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.540102 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.540315 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 02:57:02.541608 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.523668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33927 (local address 127.22.230.3:51761)
0619 02:57:02.524357 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:02.524393 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:02.526053 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:02.526778 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:02.526786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:02.527294 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:02.527826 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:02.527843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.528908 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.528912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:02.530118 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:02.530129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.530952 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.530960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:02.531180 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.531934 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:02.531969 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:02.534806 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:02.537864 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:02.537873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:02.537888 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:02.538299 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:02.538721 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:02.538726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:02.538730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:02.538913 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:02.539431 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:02.539439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:02.539773 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 02:57:02.540029 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 02:57:02.540550 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":286,"threads_started":1}
I20260619 02:57:02.542887 29520 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33927
I20260619 02:57:02.543179 29520 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:02.543856 29520 heartbeater.cc:511] Master 127.22.230.62:33927 requested a full tablet report, sending...
I20260619 02:57:02.545277 29064 ts_manager.cc:194] Registered new tserver with Master: fc5e6f75ceed435fb86634488de8a5bb (127.22.230.3:46505)
I20260619 02:57:02.546363 29064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:51761
I20260619 02:57:02.547348 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/test-admin.keytab.
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.573349 23448 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260619 02:57:02.577973 29253 heartbeater.cc:503] Master 127.22.230.62:33927 was elected leader, sending a full tablet report...
W20260619 02:57:02.579185 29527 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:02.576731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.22.230.1:45323)
0619 02:57:02.577311 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.577318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.577343 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:02.577417 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.577424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.577614 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.577742 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.578751 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36474: BlockingRecv error: recv got EOF from 127.0.0.1:36474 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":450,"thread_start_us":310,"threads_started":1}
W20260619 02:57:02.580318 23448 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.581718 23448 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.582996 23448 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:57:02.601353 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.584409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.22.230.1:45323)
0619 02:57:02.584628 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.584636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.584854 (+ 218us) server_negotiation.cc:407] Connection header received
0619 02:57:02.585027 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.585031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.585095 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.585193 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.586132 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.587173 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.588376 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.588907 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.589045 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.596215 (+ 7170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.596224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.596238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.596279 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.598846 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.599439 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.599445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.599447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.599563 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.599904 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.599909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.599912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.600237 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.600548 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.600851 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.601084 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 02:57:02.606379 29527 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:02.604922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.22.230.1:45323)
0619 02:57:02.605136 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.605141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.605304 (+ 163us) server_negotiation.cc:407] Connection header received
0619 02:57:02.605409 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.605412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.605463 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.605535 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.606115 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36486: BlockingRecv error: recv got EOF from 127.0.0.1:36486 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 02:57:02.606478 23448 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.607896 23448 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.609113 23448 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:57:02.621274 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.610342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.22.230.1:45323)
0619 02:57:02.610536 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.610541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.610751 (+ 210us) server_negotiation.cc:407] Connection header received
0619 02:57:02.610867 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.610870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.610917 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.610986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.611788 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.612406 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.613425 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.613815 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.613898 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.616967 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.616972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.616975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.616999 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.619360 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.619874 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.619877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.619879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.619976 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.620252 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.620255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.620257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.620467 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.620631 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.620856 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.621006 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260619 02:57:02.625073 29527 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:02.623575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.22.230.1:45323)
0619 02:57:02.623754 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.623760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.623954 (+ 194us) server_negotiation.cc:407] Connection header received
0619 02:57:02.624110 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.624115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.624174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.624257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.624818 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36512: BlockingRecv error: recv got EOF from 127.0.0.1:36512 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 02:57:02.625216 23448 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781837774831470-23448-0/krb5kdc/krb5cc' of type FILE
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.626538 23448 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:02.627852 23448 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:57:02.640087 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.628929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.22.230.1:45323)
0619 02:57:02.629078 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.629082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.629374 (+ 292us) server_negotiation.cc:407] Connection header received
0619 02:57:02.629466 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.629469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.629513 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.629583 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.630385 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.631074 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.632209 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.632599 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.632682 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.635788 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.635792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.635795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.635828 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.638102 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.638702 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.638705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.638707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.638819 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.639105 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.639108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.639109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.639330 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.639536 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.639718 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.639853 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:57:02.641419 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29125
I20260619 02:57:02.652387 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.642177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.22.230.1:45323)
0619 02:57:02.642352 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.642357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.642529 (+ 172us) server_negotiation.cc:407] Connection header received
0619 02:57:02.642765 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.642775 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.642824 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.642915 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.643720 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.644369 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.645523 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.645922 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.646047 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.647954 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.647964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.647967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.647992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.650054 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.650703 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.650710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.650714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.650834 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.651201 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.651206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.651209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.651428 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.651676 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.651945 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.652119 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 02:57:02.653000 29248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:02.733724 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29125
I20260619 02:57:02.749547 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29259
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:57:02.763828 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.750395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.22.230.2:33483)
0619 02:57:02.751006 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.751013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.751037 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:57:02.751093 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.751099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.751296 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.751438 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.752290 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.753283 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.754432 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.755051 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.755144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.758602 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.758623 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.758636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.758680 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.761363 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.761946 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.761952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.761954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.762081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.762398 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.762401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.762403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.762755 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.762973 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.763247 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.763498 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":340,"threads_started":1}
I20260619 02:57:02.765120 29382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:02.846858 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29259
I20260619 02:57:02.862557 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29392
Jun 19 02:57:02 dist-test-slave-2gms krb5kdc[29019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:57:02.877267 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:02.863645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.22.230.3:46505)
0619 02:57:02.864372 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:02.864378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:02.864406 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:57:02.864467 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:02.864473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:02.864680 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:02.864805 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:02.865738 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.866820 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.868129 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:02.868650 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:02.868749 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:02.872501 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:02.872510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:02.872522 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:02.872549 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:02.875065 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.875616 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.875622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.875624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.875734 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:02.876024 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:02.876027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:02.876029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:02.876333 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:02.876532 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:02.876759 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 02:57:02.876994 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":396,"threads_started":1}
I20260619 02:57:02.878429 29515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:02.959740 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29392
I20260619 02:57:02.977324 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29032
I20260619 02:57:02.978400 29094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:03.079661 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29032
2026-06-19T02:57:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3795 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:57:03 dist-test-slave-2gms krb5kdc[29547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:57:03 dist-test-slave-2gms krb5kdc[29547](info): set up 2 sockets
Jun 19 02:57:03 dist-test-slave-2gms krb5kdc[29547](info): commencing operation
krb5kdc: starting...
W20260619 02:57:04.171572 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:04 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab.
Entry for principal oryx/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab.
I20260619 02:57:04.373464 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38487
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab
--principal=oryx/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:04.683240 29560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:04.683497 29560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:04.683642 29560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:04.690667 29560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:04.690788 29560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:04.690871 29560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:04.690968 29560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:04.691051 29560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:04.700755 29560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab
--principal=oryx/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:04.702010 29560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:04.703308 29560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:04.714406 29565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:04.714496 29566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:04.714439 29568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:04.715473 29560 server_base.cc:1034] running on GCE node
I20260619 02:57:04.716231 29560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:04.717550 29560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:04.718766 29560 hybrid_clock.cc:648] HybridClock initialized: now 1781837824718730 us; error 50 us; skew 500 ppm
I20260619 02:57:04.719092 29560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:04 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:04.722258 29560 init.cc:377] Logged in from keytab as oryx/127.22.230.62@KRBTEST.COM (short username oryx)
I20260619 02:57:04.723685 29560 webserver.cc:460] Webserver started at http://127.22.230.62:35605/ using document root <none> and password file <none>
I20260619 02:57:04.724318 29560 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:04.724485 29560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:04.724781 29560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:04.727593 29560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "fe80c7302c424ac0bb2ce35787ef3f60"
format_stamp: "Formatted at 2026-06-19 02:57:04 on dist-test-slave-2gms"
server_key: "6fa6966e5ddb208012606a9f0f607c27"
server_key_iv: "794d1202e6249541de7828df89a905ca"
server_key_version: "encryptionkey@0"
I20260619 02:57:04.728293 29560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "fe80c7302c424ac0bb2ce35787ef3f60"
format_stamp: "Formatted at 2026-06-19 02:57:04 on dist-test-slave-2gms"
server_key: "6fa6966e5ddb208012606a9f0f607c27"
server_key_iv: "794d1202e6249541de7828df89a905ca"
server_key_version: "encryptionkey@0"
I20260619 02:57:04.733749 29560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:57:04.738334 29575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:04.740037 29560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 02:57:04.740214 29560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "fe80c7302c424ac0bb2ce35787ef3f60"
format_stamp: "Formatted at 2026-06-19 02:57:04 on dist-test-slave-2gms"
server_key: "6fa6966e5ddb208012606a9f0f607c27"
server_key_iv: "794d1202e6249541de7828df89a905ca"
server_key_version: "encryptionkey@0"
I20260619 02:57:04.740402 29560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:04.757850 29560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:04.761595 29560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:04.761854 29560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:04.778918 29560 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:38487
I20260619 02:57:04.778954 29626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:38487 every 8 connection(s)
I20260619 02:57:04.780187 29560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:04.781495 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29560
I20260619 02:57:04.781700 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:04.782526 23448 external_mini_cluster.cc:1428] Setting key 458cbc4477f10aaa384a40b5254a560d
I20260619 02:57:04.784368 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:04.794844 29627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Bootstrap starting.
Jun 19 02:57:04 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:04.798211 29627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:04.799245 29627 log.cc:824] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:04.802227 29627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: No bootstrap required, opened a new log
I20260619 02:57:04.803086 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:04.784504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.22.230.62:38487)
0619 02:57:04.785175 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:04.785205 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:04.785241 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:57:04.786797 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:04.786816 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:04.787185 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:04.787552 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:04.788893 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:04.790150 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:04.791373 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:04.792065 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:04.792266 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:04.796419 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:04.796426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:04.796438 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:04.796473 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:04.799299 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:04.800099 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:04.800109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:04.800120 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:04.800324 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:04.800715 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:04.800718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:04.800720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:04.801161 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:04.801379 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:04.801782 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 02:57:04.802092 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":291,"threads_started":1}
I20260619 02:57:04.809276 29627 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } }
I20260619 02:57:04.809617 29627 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:04.809721 29627 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe80c7302c424ac0bb2ce35787ef3f60, State: Initialized, Role: FOLLOWER
I20260619 02:57:04.810261 29627 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } }
I20260619 02:57:04.810488 29627 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:04.810636 29627 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:04.810804 29627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:04.812768 29627 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } }
I20260619 02:57:04.813220 29627 leader_election.cc:302] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe80c7302c424ac0bb2ce35787ef3f60; no voters:
I20260619 02:57:04.813853 29627 leader_election.cc:288] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:04.813961 29632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:04.814687 29632 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 LEADER]: Becoming Leader. State: Replica: fe80c7302c424ac0bb2ce35787ef3f60, State: Running, Role: LEADER
I20260619 02:57:04.815202 29632 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } }
I20260619 02:57:04.816025 29627 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:04.817919 29634 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe80c7302c424ac0bb2ce35787ef3f60. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } }
I20260619 02:57:04.818014 29633 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } }
I20260619 02:57:04.818341 29634 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:04.818516 29633 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:04.819800 29637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:04.824816 29637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:04.833457 29637 catalog_manager.cc:1347] Generated new cluster ID: f220dbd2e48f42848627ebe0b42712d8
I20260619 02:57:04.833562 29637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:04.846345 29637 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:04.847572 29637 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:04.853843 29637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Generated new TSK 0
I20260619 02:57:04.854746 29637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab.
Entry for principal oryx/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab.
I20260619 02:57:04.912057 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38487
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:05.226858 29653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:05.227120 29653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:05.227298 29653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:05.234045 29653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:05.234176 29653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:05.234354 29653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:05.244223 29653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:05.245486 29653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:05.246847 29653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:05.259723 29659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:05.259676 29658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:05.259663 29661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:05.260516 29653 server_base.cc:1034] running on GCE node
I20260619 02:57:05.261174 29653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:05.262048 29653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:05.263252 29653 hybrid_clock.cc:648] HybridClock initialized: now 1781837825263205 us; error 66 us; skew 500 ppm
I20260619 02:57:05.263558 29653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:05 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837825, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:05.266695 29653 init.cc:377] Logged in from keytab as oryx/127.22.230.1@KRBTEST.COM (short username oryx)
I20260619 02:57:05.267988 29653 webserver.cc:460] Webserver started at http://127.22.230.1:40859/ using document root <none> and password file <none>
I20260619 02:57:05.268628 29653 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:05.268757 29653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:05.269047 29653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:05.272017 29653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "c2d12dd94a5c44168357171a07737412"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "a1b31692e45f8d56032f9639bc840280"
server_key_iv: "6aae7ce6cc0c22793ac74f91a42174c1"
server_key_version: "encryptionkey@0"
I20260619 02:57:05.272804 29653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "c2d12dd94a5c44168357171a07737412"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "a1b31692e45f8d56032f9639bc840280"
server_key_iv: "6aae7ce6cc0c22793ac74f91a42174c1"
server_key_version: "encryptionkey@0"
I20260619 02:57:05.277688 29653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:57:05.280884 29668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:05.282047 29653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:57:05.282213 29653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "c2d12dd94a5c44168357171a07737412"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "a1b31692e45f8d56032f9639bc840280"
server_key_iv: "6aae7ce6cc0c22793ac74f91a42174c1"
server_key_version: "encryptionkey@0"
I20260619 02:57:05.282388 29653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:05.302376 29653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:05.306073 29653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:05.306344 29653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:05.307314 29653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:05.308554 29653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:05.308650 29653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:05.308796 29653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:05.308864 29653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:05.341025 29653 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:34993
I20260619 02:57:05.341070 29780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:34993 every 8 connection(s)
I20260619 02:57:05.342219 29653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:05.351468 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29653
I20260619 02:57:05.351675 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:05.352401 23448 external_mini_cluster.cc:1428] Setting key 8b993cb8ce75a77c2905bc1396ae28aa
Jun 19 02:57:05 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837825, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:05.362766 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:05.344587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:51479 (local address 127.22.230.62:38487)
0619 02:57:05.344998 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:05.345004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:05.346041 (+ 1037us) server_negotiation.cc:407] Connection header received
0619 02:57:05.347730 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:05.347734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:05.347795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:05.347886 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:05.349187 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.349856 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.351606 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.352038 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.352130 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:05.356297 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:05.356305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:05.356313 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:05.356357 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:05.359751 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:05.360448 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:05.360456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:05.360462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:05.360602 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:05.360978 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:05.360986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:05.360992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:05.361379 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:05.361529 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:05.362286 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 02:57:05.362435 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":212,"threads_started":1}
I20260619 02:57:05.364207 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:05.345207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:51479)
0619 02:57:05.345881 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:05.345924 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:05.347503 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:05.348072 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:05.348079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:05.348529 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:05.349056 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:05.349071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.349965 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.349968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:05.351440 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:05.351456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.352257 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.352266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:05.352568 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:05.353561 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:05.353599 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:05.356062 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:05.359894 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:05.359902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:05.359914 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:05.360324 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:05.360700 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:05.360703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:05.360705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:05.360871 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:05.361493 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:05.361499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:05.361890 (+ 391us) client_negotiation.cc:769] Sending connection context
0619 02:57:05.362200 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 02:57:05.362786 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":247,"threads_started":1}
I20260619 02:57:05.365660 29781 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38487
I20260619 02:57:05.365940 29781 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:05.366668 29781 heartbeater.cc:511] Master 127.22.230.62:38487 requested a full tablet report, sending...
I20260619 02:57:05.368558 29592 ts_manager.cc:194] Registered new tserver with Master: c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993)
I20260619 02:57:05.369881 29592 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:51479
WARNING: no policy specified for oryx/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab.
Entry for principal oryx/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab.
I20260619 02:57:05.398938 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38487
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:05.713361 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:05.713641 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:05.713819 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:05.721172 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:05.721299 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:05.721462 29787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:05.730753 29787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:05.731819 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:05.732997 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:05.745044 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:05.745004 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:05.745396 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:05.745970 29787 server_base.cc:1034] running on GCE node
I20260619 02:57:05.746506 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:05.747373 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:05.748587 29787 hybrid_clock.cc:648] HybridClock initialized: now 1781837825748551 us; error 49 us; skew 500 ppm
I20260619 02:57:05.748878 29787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:05 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837825, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:05.754554 29787 init.cc:377] Logged in from keytab as oryx/127.22.230.2@KRBTEST.COM (short username oryx)
I20260619 02:57:05.755952 29787 webserver.cc:460] Webserver started at http://127.22.230.2:46173/ using document root <none> and password file <none>
I20260619 02:57:05.756583 29787 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:05.756719 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:05.756994 29787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:05.759721 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "b4baead2a72f46ac962b668e105f2c7e"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "43b138e564260a57dc8a199d3e3059ec"
server_key_iv: "c551339a9d1284b6ab81cfe5ea4562cf"
server_key_version: "encryptionkey@0"
I20260619 02:57:05.760727 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "b4baead2a72f46ac962b668e105f2c7e"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "43b138e564260a57dc8a199d3e3059ec"
server_key_iv: "c551339a9d1284b6ab81cfe5ea4562cf"
server_key_version: "encryptionkey@0"
I20260619 02:57:05.768671 29787 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260619 02:57:05.773717 29802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:05.775754 29787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 02:57:05.775923 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "b4baead2a72f46ac962b668e105f2c7e"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "43b138e564260a57dc8a199d3e3059ec"
server_key_iv: "c551339a9d1284b6ab81cfe5ea4562cf"
server_key_version: "encryptionkey@0"
I20260619 02:57:05.776252 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:05.800527 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:05.805585 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:05.805873 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:05.806903 29787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:05.808418 29787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:05.808516 29787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:05.808630 29787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:05.808712 29787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:05.843946 29787 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:34311
I20260619 02:57:05.843966 29914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:34311 every 8 connection(s)
I20260619 02:57:05.845198 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:05.847512 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29787
I20260619 02:57:05.847699 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:05.848238 23448 external_mini_cluster.cc:1428] Setting key 699b12cf4e0c207df6a033b7141a73c6
Jun 19 02:57:05 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837825, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:05.867328 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:05.847815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:37065 (local address 127.22.230.62:38487)
0619 02:57:05.847981 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:05.847985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:05.849374 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 02:57:05.851539 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:05.851547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:05.851633 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:05.851744 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:05.853071 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.854288 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.855998 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.856735 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.856886 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:05.860500 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:05.860509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:05.860518 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:05.860559 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:05.864012 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:05.864666 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:05.864674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:05.864680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:05.864844 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:05.865410 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:05.865418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:05.865424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:05.865794 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:05.865942 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:05.866634 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 02:57:05.866877 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:57:05.868355 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:05.848404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:37065)
0619 02:57:05.849236 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:05.849273 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:05.851288 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:05.851918 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:05.851925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:05.852410 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:05.852913 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:05.852929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.854434 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.854437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:05.855862 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:05.855873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:05.856952 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:05.856956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:05.857294 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:05.858004 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:05.858046 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:05.860291 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:05.864137 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:05.864143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:05.864154 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:05.864529 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:05.864999 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:05.865003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:05.865006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:05.865272 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:05.865922 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:05.865928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:05.866229 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 02:57:05.866440 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 02:57:05.866874 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":340,"threads_started":1}
I20260619 02:57:05.869637 29915 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38487
I20260619 02:57:05.869928 29915 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:05.870644 29915 heartbeater.cc:511] Master 127.22.230.62:38487 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:57:05.872155 29592 ts_manager.cc:194] Registered new tserver with Master: b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:05.873018 29592 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:37065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab.
Entry for principal oryx/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab.
I20260619 02:57:05.895879 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38487
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:06.202661 29920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:06.202955 29920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:06.203128 29920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:06.210064 29920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:06.210230 29920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:06.210407 29920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:06.219837 29920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:06.221043 29920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:06.222283 29920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:06.235071 29928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:06.235219 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:06.235260 29926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:06.236079 29920 server_base.cc:1034] running on GCE node
I20260619 02:57:06.236691 29920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:06.237534 29920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:06.238703 29920 hybrid_clock.cc:648] HybridClock initialized: now 1781837826238645 us; error 49 us; skew 500 ppm
I20260619 02:57:06.238987 29920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837826, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:06.241897 29920 init.cc:377] Logged in from keytab as oryx/127.22.230.3@KRBTEST.COM (short username oryx)
I20260619 02:57:06.243325 29920 webserver.cc:460] Webserver started at http://127.22.230.3:43117/ using document root <none> and password file <none>
I20260619 02:57:06.244038 29920 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:06.244179 29920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:06.244464 29920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:06.247290 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "48bad7dcc83b4d3d9f455747f7b85970"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "2211a4d76494cd807084288991d6a5b3"
server_key_iv: "d6a8acd570d977870382262b8ddc9b5d"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.248024 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "48bad7dcc83b4d3d9f455747f7b85970"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "2211a4d76494cd807084288991d6a5b3"
server_key_iv: "d6a8acd570d977870382262b8ddc9b5d"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.252825 29920 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 02:57:06.256163 29935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:06.257495 29920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 02:57:06.257659 29920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "48bad7dcc83b4d3d9f455747f7b85970"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "2211a4d76494cd807084288991d6a5b3"
server_key_iv: "d6a8acd570d977870382262b8ddc9b5d"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.257838 29920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:06.277256 29920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:06.280793 29920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:06.281077 29920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:06.282044 29920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:06.283284 29920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:06.283350 29920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:06.283493 29920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:06.283540 29920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:06.315435 29920 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40529
I20260619 02:57:06.315505 30047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40529 every 8 connection(s)
I20260619 02:57:06.316773 29920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:06.322907 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 29920
I20260619 02:57:06.323101 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:06.323653 23448 external_mini_cluster.cc:1428] Setting key 083b8efd4ebee7aa5aae02a3bbfc8f99
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837826, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:06.335958 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.319330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:39701 (local address 127.22.230.62:38487)
0619 02:57:06.319569 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.319575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.320668 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 02:57:06.322653 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.322661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.322754 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.322833 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.324633 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.325556 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.327280 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.327674 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.327805 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.330891 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.330896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.330899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.330927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.333202 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.333816 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.333823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.333828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.333939 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.334333 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.334340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.334343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.334650 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.334786 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.335410 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.335619 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260619 02:57:06.336833 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.319802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:39701)
0619 02:57:06.320465 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.320524 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.322335 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.323130 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.323142 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:06.323821 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:06.324458 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.324474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.325724 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.325736 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.327142 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.327154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.327897 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.327906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.328134 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.328702 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:06.328725 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:06.330697 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:06.333340 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.333348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.333361 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.333711 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.334036 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.334042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.334045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.334231 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.334763 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:06.334770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:06.335059 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 02:57:06.335333 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 02:57:06.335804 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260619 02:57:06.338218 30048 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38487
I20260619 02:57:06.338533 30048 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:06.339210 30048 heartbeater.cc:511] Master 127.22.230.62:38487 requested a full tablet report, sending...
I20260619 02:57:06.340667 29592 ts_manager.cc:194] Registered new tserver with Master: 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529)
I20260619 02:57:06.341416 29592 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:39701
I20260619 02:57:06.350437 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.230.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.61@KRBTEST.COM" created.
I20260619 02:57:06.372730 29781 heartbeater.cc:503] Master 127.22.230.62:38487 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.61.keytab.
Entry for principal oryx/127.22.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.61.keytab.
I20260619 02:57:06.395897 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:43577
--webserver_interface=127.22.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.61.keytab
--principal=oryx/127.22.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38487,127.22.230.61:43577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:06.710932 30053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:06.711210 30053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:06.711393 30053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:06.718674 30053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:06.718791 30053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:06.718885 30053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:06.718990 30053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:06.719066 30053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:06.728789 30053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38487,127.22.230.61:43577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.61.keytab
--principal=oryx/127.22.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:43577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
W20260619 02:57:06.729139 30053 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.230.62:38487,127.22.230.61:43577'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 02:57:06.730116 30053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:06.731396 30053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:06.742376 30058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:06.742344 30059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:06.742306 30061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:06.743393 30053 server_base.cc:1034] running on GCE node
I20260619 02:57:06.744212 30053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:06.745555 30053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:06.746781 30053 hybrid_clock.cc:648] HybridClock initialized: now 1781837826746704 us; error 56 us; skew 500 ppm
I20260619 02:57:06.747104 30053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837826, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:06.750030 30053 init.cc:377] Logged in from keytab as oryx/127.22.230.61@KRBTEST.COM (short username oryx)
I20260619 02:57:06.751433 30053 webserver.cc:460] Webserver started at http://127.22.230.61:37575/ using document root <none> and password file <none>
I20260619 02:57:06.752090 30053 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:06.752220 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:06.752513 30053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:06.755432 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/instance:
uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "0c0fa4d88aa13fcc1d899df9e0da46cd"
server_key_iv: "f4a4cac307255234e9d2f9db7522bbdd"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.756187 30053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal/instance:
uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "0c0fa4d88aa13fcc1d899df9e0da46cd"
server_key_iv: "f4a4cac307255234e9d2f9db7522bbdd"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.761179 30053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 02:57:06.764511 30068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:06.765779 30053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:57:06.765947 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "0c0fa4d88aa13fcc1d899df9e0da46cd"
server_key_iv: "f4a4cac307255234e9d2f9db7522bbdd"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.766150 30053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:06.783659 30053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:06.787739 30053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:06.787984 30053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:06.805560 30053 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.61:43577
I20260619 02:57:06.805583 30119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.61:43577 every 8 connection(s)
I20260619 02:57:06.806869 30053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
I20260619 02:57:06.810289 30120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:06.813932 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 30053
I20260619 02:57:06.818044 30120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837826, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.61@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:06.836062 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.820839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.22.230.62:38487)
0619 02:57:06.821046 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.821052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.821563 (+ 511us) server_negotiation.cc:407] Connection header received
0619 02:57:06.821744 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.821748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.821804 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.821913 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.822987 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.823684 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.825070 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.825476 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.825608 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.829283 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.829289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.829293 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.829375 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.832992 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.833831 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.833836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.833839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.833983 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.834373 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.834378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.834381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.834787 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.834928 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.835553 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.835754 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 02:57:06.836941 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.815977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.22.230.61:43577)
0619 02:57:06.816786 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.816797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.816834 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:57:06.818504 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.818524 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.818952 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.819293 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.820856 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.822617 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.823813 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.824441 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.824638 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.828865 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.828872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.828882 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.828912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.832828 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.833658 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.833664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.833667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.833921 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.834359 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.834364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.834367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.835002 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.835221 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.835591 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.835862 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":351,"threads_started":1}
I20260619 02:57:06.837074 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.820700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42236)
0619 02:57:06.821438 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.821476 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.821630 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.821978 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.821984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:06.822559 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:06.822855 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.822864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.823815 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.823821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.824920 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.824931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.825680 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.825686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.825829 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.826418 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:06.826437 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:06.829086 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:06.833118 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.833128 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.833141 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.833685 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.834089 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.834094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.834097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.834274 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.834906 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:06.834912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:06.835136 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 02:57:06.835437 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 02:57:06.836109 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":224,"thread_start_us":266,"threads_started":1}
I20260619 02:57:06.839351 30120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837826, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.61@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
I20260619 02:57:06.852310 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.840140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53588)
0619 02:57:06.840289 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.840303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.840381 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.840616 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.840619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:06.840884 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:06.841001 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.841008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.841892 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.841895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.842790 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.842800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.843667 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.843671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.843823 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.844452 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:06.844484 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:06.846415 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:06.850090 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.850094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.850098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.850443 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.850980 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.850984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.850986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.851111 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.851627 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:06.851631 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:06.851752 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 02:57:06.851868 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:57:06.852068 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260619 02:57:06.852516 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.840016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.22.230.61:43577)
0619 02:57:06.840211 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.840216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.840367 (+ 151us) server_negotiation.cc:407] Connection header received
0619 02:57:06.840465 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.840468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.840517 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.840587 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.841103 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.841782 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.842923 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.843547 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.843663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.846621 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.846627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.846632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.846668 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.849934 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.850707 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.850712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.850715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.850835 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.851213 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.851216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.851218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.851485 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.851594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.851950 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.852186 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":80}
I20260619 02:57:06.856748 30120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Bootstrap starting.
I20260619 02:57:06.859647 30120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:06.860639 30120 log.cc:824] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:06.863179 30120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: No bootstrap required, opened a new log
I20260619 02:57:06.869756 30120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } }
I20260619 02:57:06.870055 30120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:06.870141 30120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab12b4d6fb741a0aab3f5ee7bda9689, State: Initialized, Role: FOLLOWER
I20260619 02:57:06.870724 30120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } }
I20260619 02:57:06.871480 30126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } } }
I20260619 02:57:06.871817 30126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:57:06.872393 30120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:06.875622 29915 heartbeater.cc:503] Master 127.22.230.62:38487 was elected leader, sending a full tablet report...
W20260619 02:57:06.881462 30137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:57:06.881587 30137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:57:06.883437 30053 master_runner.cc:428] Detected that this master 7ab12b4d6fb741a0aab3f5ee7bda9689 is joining an existing cluster
I20260619 02:57:06.883513 30053 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.230.61:43577
I20260619 02:57:06.903272 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.887818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.22.230.62:38487)
0619 02:57:06.887976 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.887983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.888002 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:57:06.888050 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.888054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.888119 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.888234 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.889488 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.890531 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.892734 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.893482 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.893586 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.896365 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.896373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.896381 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.896422 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.899796 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.900685 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.900692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.900697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.900845 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.901333 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.901341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.901347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.901719 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.901839 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.902510 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.902792 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:57:06.904392 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.887220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.22.230.61:43577)
0619 02:57:06.887635 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.887642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.889118 (+ 1476us) server_negotiation.cc:407] Connection header received
0619 02:57:06.889279 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.889286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.889366 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.889462 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.890745 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.891847 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.893703 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.894404 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.894497 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.896941 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.896950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.896956 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.896996 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.900884 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.901740 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.901745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.901749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.901867 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.902355 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.902360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.902363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.902724 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.902852 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.903650 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.903997 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":253}
I20260619 02:57:06.904438 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.887121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42252)
0619 02:57:06.887800 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.887817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.887935 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.888254 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.888258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:06.888937 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:06.889362 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.889371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.890726 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.890731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.892562 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.892579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.893867 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.893876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.894080 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.894830 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:06.894860 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:06.896151 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:06.899918 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.899925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.899932 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.900530 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.901006 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.901012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.901018 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.901198 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.901931 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:06.901938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:06.902201 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 02:57:06.903454 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0619 02:57:06.903848 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1,"thread_start_us":1200,"threads_started":1}
I20260619 02:57:06.906447 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.887353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53600)
0619 02:57:06.888991 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.889013 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.889143 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.889598 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.889602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:06.890231 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:06.890602 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.890620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.892074 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.892078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.893568 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.893581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.894793 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.894800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.894996 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.895598 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:06.895620 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:06.896752 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:06.901030 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.901034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.901038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.901616 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.902005 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.902009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.902013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.902223 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.902933 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:06.902937 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:06.903122 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 02:57:06.903762 (+ 640us) client_negotiation.cc:240] Negotiation successful
0619 02:57:06.905986 (+ 2224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"thread_start_us":1338,"threads_started":1}
WARNING: no policy specified for oryx/127.22.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.60@KRBTEST.COM" created.
Jun 19 02:57:06 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.62@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
I20260619 02:57:06.932317 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.911113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53614)
0619 02:57:06.915082 (+ 3969us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.915128 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.915352 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.915905 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.915913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:06.916773 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:06.917045 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.917057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.918096 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.918103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.919688 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:06.919703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.921033 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.921042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:06.921279 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.921939 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:06.921971 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:06.924803 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:06.929180 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.929191 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.929199 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.929718 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.930147 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:06.930151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:06.930155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:06.930318 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:06.931032 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:06.931043 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:06.931250 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 02:57:06.931467 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 02:57:06.931801 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3742,"thread_start_us":3599,"threads_started":1}
I20260619 02:57:06.932340 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.911163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.22.230.61:43577)
0619 02:57:06.911313 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.911319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.915211 (+ 3892us) server_negotiation.cc:407] Connection header received
0619 02:57:06.915569 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.915576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.915730 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.915840 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:06.917192 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.917951 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.919836 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:06.920678 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:06.920810 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:06.924990 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:06.924997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:06.925005 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:06.925045 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:06.929048 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.929863 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.929871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.929877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.930026 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:06.930436 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:06.930444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:06.930450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:06.930869 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:06.931029 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:06.931601 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 02:57:06.931854 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:57:06.934847 29592 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.230.61:43577: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
member_type: NON_VOTER
last_known_addr {
host: "127.22.230.61"
port: 43577
}
attrs {
promote: true
}
}
dest_uuid: "fe80c7302c424ac0bb2ce35787ef3f60"
cas_config_opid_index: -1
I20260619 02:57:06.936028 29592 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } }
I20260619 02:57:06.939702 30150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:06.940534 30095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:06.944410 30150 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.230.61:43577
I20260619 02:57:06.944780 30095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Refusing update from remote peer fe80c7302c424ac0bb2ce35787ef3f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 02:57:06.945077 30053 master.cc:537] Master@127.22.230.61:43577 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.60.keytab.
Entry for principal oryx/127.22.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.60.keytab.
I20260619 02:57:06.947309 30149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:06.948362 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:38553
--webserver_interface=127.22.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.60.keytab
--principal=oryx/127.22.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:57:06.948385 30151 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:06.948706 30151 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
W20260619 02:57:06.957001 29577 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 02:57:06.959051 29577 consensus_peers.cc:487] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 -> Peer 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61:43577): Couldn't send request to peer 7ab12b4d6fb741a0aab3f5ee7bda9689. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:57:06.959604 30053 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:06.960023 30053 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:06.960196 30053 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: stopping tablet replica
I20260619 02:57:06.977104 30053 master.cc:552] Master@127.22.230.61:43577 shutdown complete.
I20260619 02:57:06.977193 30053 master_runner.cc:305] Clearing existing system tablet
I20260619 02:57:06.977582 30053 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:06.977675 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:06.980615 30053 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:57:06.982255 30156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:06.982592 30053 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:06.982995 30053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 02:57:06.983120 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "0c0fa4d88aa13fcc1d899df9e0da46cd"
server_key_iv: "f4a4cac307255234e9d2f9db7522bbdd"
server_key_version: "encryptionkey@0"
I20260619 02:57:06.988047 30053 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:06.994323 30053 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:57:06.994546 30053 log.cc:1196] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 02:57:06.995034 30053 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Deleting consensus metadata
I20260619 02:57:06.995337 30053 master_runner.cc:315] Copying system tablet from 127.22.230.62:38487
I20260619 02:57:06.997825 30053 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: tablet copy: Beginning tablet copy session from remote peer at address 127.22.230.62:38487
I20260619 02:57:07.014353 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.998672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.22.230.62:38487)
0619 02:57:06.998841 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:06.998847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:06.999309 (+ 462us) server_negotiation.cc:407] Connection header received
0619 02:57:06.999569 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:06.999575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:06.999673 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:06.999745 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.000911 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.001975 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.003947 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.004583 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.004710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.007274 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.007283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.007290 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.007337 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.010938 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.011736 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.011743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.011747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.011899 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.012373 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.012380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.012385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.012808 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.012956 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.013606 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.013911 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:57:07.015424 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:06.998626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42264)
0619 02:57:06.999196 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:06.999223 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:06.999418 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:06.999935 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:06.999940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.000444 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.000721 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.000736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.002151 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.002156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.003782 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.003795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.004946 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.004954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.005127 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.005862 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.005884 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.007047 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.011102 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.011107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.011111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.011582 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.012038 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.012042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.012045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.012217 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.013050 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.013056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.013276 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.014647 (+ 1371us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.014980 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":319,"threads_started":1}
I20260619 02:57:07.016310 29612 tablet_copy_service.cc:140] P fe80c7302c424ac0bb2ce35787ef3f60: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7ab12b4d6fb741a0aab3f5ee7bda9689 ({username='oryx', principal='oryx/127.22.230.61@KRBTEST.COM'} at 127.0.0.1:42264)
I20260619 02:57:07.016631 29612 tablet_copy_service.cc:161] P fe80c7302c424ac0bb2ce35787ef3f60: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7ab12b4d6fb741a0aab3f5ee7bda9689 at {username='oryx', principal='oryx/127.22.230.61@KRBTEST.COM'} at 127.0.0.1:42264: session id = 7ab12b4d6fb741a0aab3f5ee7bda9689-00000000000000000000000000000000
I20260619 02:57:07.021246 29612 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:57:07.023488 30053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:57:07.029984 30053 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: tablet copy: Starting download of 0 data blocks...
I20260619 02:57:07.030259 30053 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: tablet copy: Starting download of 1 WAL segments...
I20260619 02:57:07.032819 30053 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:57:07.037223 29612 tablet_copy_service.cc:342] P fe80c7302c424ac0bb2ce35787ef3f60: Request end of tablet copy session 7ab12b4d6fb741a0aab3f5ee7bda9689-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.230.61@KRBTEST.COM'} at 127.0.0.1:42264
I20260619 02:57:07.037403 29612 tablet_copy_service.cc:433] P fe80c7302c424ac0bb2ce35787ef3f60: ending tablet copy session 7ab12b4d6fb741a0aab3f5ee7bda9689-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7ab12b4d6fb741a0aab3f5ee7bda9689
I20260619 02:57:07.048813 30053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:07.051281 30163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:07.052475 30164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:07.058753 30053 server_base.cc:1034] running on GCE node
W20260619 02:57:07.059100 30167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:07.059526 30053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:07.060113 30053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:07.061319 30053 hybrid_clock.cc:648] HybridClock initialized: now 1781837827061274 us; error 49 us; skew 500 ppm
I20260619 02:57:07.061609 30053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837827, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:07.063884 30053 init.cc:377] Logged in from keytab as oryx/127.22.230.61@KRBTEST.COM (short username oryx)
I20260619 02:57:07.065065 30053 webserver.cc:460] Webserver started at http://127.22.230.61:35789/ using document root <none> and password file <none>
I20260619 02:57:07.065436 30053 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:07.065514 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:07.068528 30053 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 02:57:07.073988 30173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:07.074471 30053 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.005s
I20260619 02:57:07.074633 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "0c0fa4d88aa13fcc1d899df9e0da46cd"
server_key_iv: "f4a4cac307255234e9d2f9db7522bbdd"
server_key_version: "encryptionkey@0"
I20260619 02:57:07.074795 30053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:07.095661 30053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:07.099617 30053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:07.123466 30053 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.61:43577
I20260619 02:57:07.123520 30224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.61:43577 every 8 connection(s)
I20260619 02:57:07.124498 30053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
I20260619 02:57:07.129707 30225 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:57:07.133585 30225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Bootstrap starting.
I20260619 02:57:07.148674 30225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:57:07.149214 30225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Bootstrap complete.
I20260619 02:57:07.150209 30225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } }
I20260619 02:57:07.150458 30225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ab12b4d6fb741a0aab3f5ee7bda9689, State: Initialized, Role: LEARNER
I20260619 02:57:07.150714 30225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } }
I20260619 02:57:07.151816 30225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:07.151654 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:07.152174 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:57:07.162586 30239 catalog_manager.cc:1259] Loaded cluster ID: f220dbd2e48f42848627ebe0b42712d8
I20260619 02:57:07.162695 30239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: loading cluster ID for follower catalog manager: success
I20260619 02:57:07.166550 30239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: acquiring CA information for follower catalog manager: success
I20260619 02:57:07.169162 30239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:57:07.331375 30153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:07.331679 30153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:07.331821 30153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:07.339586 30153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:07.339738 30153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:07.339839 30153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:07.339938 30153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:07.340036 30153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:07.343751 30048 heartbeater.cc:503] Master 127.22.230.62:38487 was elected leader, sending a full tablet report...
I20260619 02:57:07.350001 30153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.60.keytab
--principal=oryx/127.22.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:38553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:07.351267 30153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:07.352545 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:07.364334 30247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:07.364379 30245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:07.364403 30244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:07.365250 30153 server_base.cc:1034] running on GCE node
I20260619 02:57:07.365912 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:07.367274 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:07.368497 30153 hybrid_clock.cc:648] HybridClock initialized: now 1781837827368462 us; error 47 us; skew 500 ppm
I20260619 02:57:07.368822 30153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837827, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:07.371711 30153 init.cc:377] Logged in from keytab as oryx/127.22.230.60@KRBTEST.COM (short username oryx)
I20260619 02:57:07.373450 30153 webserver.cc:460] Webserver started at http://127.22.230.60:46801/ using document root <none> and password file <none>
I20260619 02:57:07.374048 30153 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:07.374183 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:07.374470 30153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:07.377655 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/instance:
uuid: "0bb2a01f104647df976f364cdd0f0dd2"
format_stamp: "Formatted at 2026-06-19 02:57:07 on dist-test-slave-2gms"
server_key: "f7019779c18221e9200247aa5236993d"
server_key_iv: "14e0c55e7c245ad8c0d8e4e134e88e66"
server_key_version: "encryptionkey@0"
I20260619 02:57:07.378312 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal/instance:
uuid: "0bb2a01f104647df976f364cdd0f0dd2"
format_stamp: "Formatted at 2026-06-19 02:57:07 on dist-test-slave-2gms"
server_key: "f7019779c18221e9200247aa5236993d"
server_key_iv: "14e0c55e7c245ad8c0d8e4e134e88e66"
server_key_version: "encryptionkey@0"
I20260619 02:57:07.383152 30153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:57:07.386358 30254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:07.387620 30153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:07.387779 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
uuid: "0bb2a01f104647df976f364cdd0f0dd2"
format_stamp: "Formatted at 2026-06-19 02:57:07 on dist-test-slave-2gms"
server_key: "f7019779c18221e9200247aa5236993d"
server_key_iv: "14e0c55e7c245ad8c0d8e4e134e88e66"
server_key_version: "encryptionkey@0"
I20260619 02:57:07.387950 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:07.399401 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:07.403580 30153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:07.403779 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:07.422729 30153 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.60:38553
I20260619 02:57:07.422878 30305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.60:38553 every 8 connection(s)
I20260619 02:57:07.424043 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
I20260619 02:57:07.425963 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 30153
I20260619 02:57:07.430374 30306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:07.438557 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.429070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53630)
0619 02:57:07.429314 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.429331 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.429431 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.432709 (+ 3278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.432713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.432744 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:07.432945 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.432955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.434907 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.434913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.436527 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.436538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.437764 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.437770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.437961 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.438013 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.438107 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.438243 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 02:57:07.438884 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.429199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.22.230.61:43577)
0619 02:57:07.432316 (+ 3117us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.432322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.432341 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:57:07.432409 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.432416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.432532 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.432642 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:07.433135 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.434755 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.436671 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.437587 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.437745 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.437960 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.438157 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.438292 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2947,"thread_start_us":2842,"threads_started":1}
I20260619 02:57:07.439921 30200 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 02:57:07.439638 30306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:57:07.440646 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe80c7302c424ac0bb2ce35787ef3f60. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:07.440856 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: LEARNER
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.60@KRBTEST.COM
I20260619 02:57:07.449505 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.428055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.22.230.60:38553)
0619 02:57:07.428744 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.428755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.428822 (+ 67us) server_negotiation.cc:407] Connection header received
0619 02:57:07.430708 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.430728 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.431063 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.431423 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.433072 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.434351 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.436074 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.437109 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.437291 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.442013 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.442025 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.442052 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.442103 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.445709 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.446428 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.446434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.446439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.446673 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.447066 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.447074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.447077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.447528 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.447811 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.448248 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.448604 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":319,"threads_started":1}
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837827, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.60@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:07.456485 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.442286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42274)
0619 02:57:07.443156 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.443186 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.443351 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.443662 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.443668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.444903 (+ 1235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.445265 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.445274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.446504 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.446508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.447642 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.447652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.448609 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.448615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.448743 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.449288 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.449303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.451567 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.454355 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.454362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.454367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.454832 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.455183 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.455186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.455188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.455299 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.455766 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.455771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.455891 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.456023 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.456227 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":408,"threads_started":1}
I20260619 02:57:07.456553 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.442301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.22.230.62:38487)
0619 02:57:07.442432 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.442435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.443267 (+ 832us) server_negotiation.cc:407] Connection header received
0619 02:57:07.443441 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.443447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.443525 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.443616 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.445389 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.446313 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.447832 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.448288 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.448355 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.451736 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.451742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.451746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.451779 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.454244 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.454931 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.454934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.454936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.455068 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.455418 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.455421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.455423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.455657 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.455758 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.456178 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.456324 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 02:57:07.457703 30306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837827, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.60@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
I20260619 02:57:07.469450 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.458553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53638)
0619 02:57:07.458751 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.458769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.458857 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.459167 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.459171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.459498 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.459614 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.459621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.460608 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.460611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.461541 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.461551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.462213 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.462218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.462390 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.462852 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.462867 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.464583 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.467738 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.467741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.467743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.468070 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.468387 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.468390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.468392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.468500 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.468888 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.468891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.468984 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.469094 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.469258 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:57:07.469578 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.458631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.22.230.61:43577)
0619 02:57:07.458795 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.458803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.458821 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:57:07.458987 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.458991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.459059 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.459153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.459750 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.460444 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.461678 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.462076 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.462203 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.464728 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.464732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.464735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.464762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.467637 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.468170 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.468173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.468175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.468284 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.468582 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.468585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.468587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.468802 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.468900 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.469168 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.469310 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:57:07.470422 30306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837827, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.60@KRBTEST.COM for oryx/127.22.230.60@KRBTEST.COM
I20260619 02:57:07.481302 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.471131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.0.0.1:43916)
0619 02:57:07.471270 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.471282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.471351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.471612 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.471615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.471873 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.471990 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.471996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.472847 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.472850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.473627 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.473636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.474343 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.474352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.474548 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.475215 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.475229 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.477063 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.479396 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.479399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.479401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.479702 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.480024 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.480027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.480029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.480151 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.480530 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.480532 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.480652 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.480742 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.480904 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:57:07.481395 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.471077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.22.230.60:38553)
0619 02:57:07.471217 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.471221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.471335 (+ 114us) server_negotiation.cc:407] Connection header received
0619 02:57:07.471419 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.471424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.471488 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.471593 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.472125 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.472752 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.473742 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.474155 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.474239 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.477201 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.477205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.477208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.477232 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.479290 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.479788 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.479791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.479793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.479927 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.480232 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.480235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.480237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.480445 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.480538 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.480832 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.481126 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":50}
I20260619 02:57:07.485841 30306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Bootstrap starting.
I20260619 02:57:07.489187 30306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:07.490115 30306 log.cc:824] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:07.492492 30306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: No bootstrap required, opened a new log
I20260619 02:57:07.499178 30306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } }
I20260619 02:57:07.499459 30306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:07.499563 30306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb2a01f104647df976f364cdd0f0dd2, State: Initialized, Role: FOLLOWER
I20260619 02:57:07.500046 30306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } }
I20260619 02:57:07.500790 30313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } } }
I20260619 02:57:07.501536 30313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:57:07.502118 30306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 02:57:07.510953 30324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:57:07.511093 30324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:57:07.515080 30153 master_runner.cc:428] Detected that this master 0bb2a01f104647df976f364cdd0f0dd2 is joining an existing cluster
I20260619 02:57:07.515187 30153 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.230.60:38553
I20260619 02:57:07.537236 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.519912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.22.230.62:38487)
0619 02:57:07.520307 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.520315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.520804 (+ 489us) server_negotiation.cc:407] Connection header received
0619 02:57:07.520879 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.520887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.520982 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.521103 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.522741 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.524162 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.526866 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.527507 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.527597 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.530627 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.530633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.530638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.530675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.534091 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.534866 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.534874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.534883 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.535034 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.535549 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.535556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.535561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.535896 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.536023 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.536660 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.536872 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260619 02:57:07.538708 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.519063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42276)
0619 02:57:07.520568 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.520588 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.520713 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.521148 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.521155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.522005 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.522524 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.522539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.524359 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.524364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.526692 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.526709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.527940 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.527946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.528140 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.529340 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.529362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.530428 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.534217 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.534221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.534224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.534717 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.535184 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.535192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.535198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.535406 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.536118 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.536125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.536339 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.537458 (+ 1119us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.537873 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":41,"thread_start_us":1195,"threads_started":1}
I20260619 02:57:07.541069 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.520790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.22.230.60:38553)
0619 02:57:07.520951 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.520961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.525438 (+ 4477us) server_negotiation.cc:407] Connection header received
0619 02:57:07.525647 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.525652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.525709 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.525786 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.527332 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.528553 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.530892 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.531549 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.531638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.534356 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.534362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.534367 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.534399 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.538318 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.539077 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.539082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.539086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.539241 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.539617 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.539624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.539630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.539934 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.540075 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.540543 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.540733 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":32}
I20260619 02:57:07.541608 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.519912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.22.230.61:43577)
0619 02:57:07.520140 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.520149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.524535 (+ 4386us) server_negotiation.cc:407] Connection header received
0619 02:57:07.525321 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.525327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.525396 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.525463 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.527010 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.527958 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.530419 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.531077 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.531179 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.534263 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.534270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.534277 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.534332 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.538036 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.539424 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.539428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.539430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.539561 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.540075 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.540083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.540085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.540385 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.540502 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.541046 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.541249 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:57:07.541950 30332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.521034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.0.0.1:43922)
0619 02:57:07.525352 (+ 4318us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.525369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.525525 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.525970 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.525974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.526867 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.527194 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.527207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.528736 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.528741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.530734 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.530747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.531807 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.531814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.532035 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.532830 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.532904 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.534171 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.538458 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.538462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.538466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.538967 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.539362 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.539366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.539369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.539517 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.540135 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.540140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.540286 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.541217 (+ 931us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.541510 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4190,"spinlock_wait_cycles":17664,"thread_start_us":4049,"threads_started":1}
I20260619 02:57:07.543227 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.519612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53642)
0619 02:57:07.524427 (+ 4815us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.524444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.524917 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.525654 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.525658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.526465 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.526835 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.526847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.528147 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.528151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.530261 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.530275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.531471 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.531479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.531860 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.532806 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.532830 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.534077 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.538435 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.538440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.538444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.539296 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.539693 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.539697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.539700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.539934 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.540580 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.540584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.540769 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.542534 (+ 1765us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.542846 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4661,"mutex_wait_us":26,"thread_start_us":4103,"threads_started":1}
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.62@KRBTEST.COM for oryx/127.22.230.60@KRBTEST.COM
I20260619 02:57:07.555089 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29653
I20260619 02:57:07.556802 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.544515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.0.0.1:43938)
0619 02:57:07.544734 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.544750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.544851 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.545181 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.545188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.545532 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.545670 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.545677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.546767 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.546773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.547618 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.547628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.548357 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.548363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.548519 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.549034 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.549049 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.551189 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.554251 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.554258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.554263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.554737 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.555067 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.555073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.555076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.555244 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.555883 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.555887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.556029 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.556155 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.556423 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260619 02:57:07.557726 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.544561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.22.230.60:38553)
0619 02:57:07.544742 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.544750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.544796 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:57:07.544979 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.544982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.545055 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.545165 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.545802 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.546655 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.547747 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.548194 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.548317 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.551355 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.551360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.551362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.551387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.554075 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.554836 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.554841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.554844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.554956 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.555383 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.555387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.555391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.555751 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.557032 (+ 1281us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.557137 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.557328 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 02:57:07.559005 29592 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.230.60:38553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2"
member_type: NON_VOTER
last_known_addr {
host: "127.22.230.60"
port: 38553
}
attrs {
promote: true
}
}
dest_uuid: "fe80c7302c424ac0bb2ce35787ef3f60"
cas_config_opid_index: 5
I20260619 02:57:07.560014 29592 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } }
I20260619 02:57:07.562327 30200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 LEARNER]: Refusing update from remote peer fe80c7302c424ac0bb2ce35787ef3f60: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:57:07.563458 30281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:07.563388 30151 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.567703 30281 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Refusing update from remote peer fe80c7302c424ac0bb2ce35787ef3f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 02:57:07.571794 30335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 02:57:07.572476 30335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:07.577333 30151 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.230.60:38553
I20260619 02:57:07.577801 30153 master.cc:537] Master@127.22.230.60:38553 shutting down...
I20260619 02:57:07.578488 30334 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.578876 30334 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.1@KRBTEST.COM
I20260619 02:57:07.590492 30200 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.596194 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.596450 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.555997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.22.230.1:34993)
0619 02:57:07.556735 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.556743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.556771 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:57:07.556841 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.556853 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.557252 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.558998 (+ 1746us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.559272 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.560789 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.570311 (+ 9522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.571818 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.573451 (+ 1633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.588573 (+ 15122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.588589 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.588605 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.588646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.592431 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.593244 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.593253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.593256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.593406 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.593880 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.593886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.593889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.594294 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.594602 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.595795 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.596044 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":398,"threads_started":1}
I20260619 02:57:07.596508 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:57:07.597343 30336 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: attempting to promote NON_VOTER 7ab12b4d6fb741a0aab3f5ee7bda9689 to VOTER
I20260619 02:57:07.597782 29776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:07.598444 30336 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } }
I20260619 02:57:07.600858 30281 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } }
W20260619 02:57:07.601044 30281 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } }
I20260619 02:57:07.605835 30200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 LEARNER]: Refusing update from remote peer fe80c7302c424ac0bb2ce35787ef3f60: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260619 02:57:07.606089 29579 consensus_peers.cc:487] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 -> Peer 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60:38553): Couldn't send request to peer 0bb2a01f104647df976f364cdd0f0dd2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 02:57:07.606408 30334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 02:57:07.608866 30151 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.612851 30200 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.616163 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.616405 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:57:07.616910 30336 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.617216 30336 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:07.617581 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.617847 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:07.618919 30281 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:07.623188 30346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:07.623612 30346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 02:57:07.624006 30313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe80c7302c424ac0bb2ce35787ef3f60. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:07.624176 30313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 02:57:07.624433 30343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: NON_VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: true } } }
I20260619 02:57:07.624625 30343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 02:57:07.625866 30153 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 02:57:07.626219 30153 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 02:57:07.626394 30153 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: stopping tablet replica
I20260619 02:57:07.633133 30153 master.cc:552] Master@127.22.230.60:38553 shutdown complete.
I20260619 02:57:07.633209 30153 master_runner.cc:305] Clearing existing system tablet
I20260619 02:57:07.633638 30153 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:07.633746 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:07.639171 30153 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:57:07.640812 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:07.641156 30153 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:07.641382 30153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 02:57:07.641472 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
uuid: "0bb2a01f104647df976f364cdd0f0dd2"
format_stamp: "Formatted at 2026-06-19 02:57:07 on dist-test-slave-2gms"
server_key: "f7019779c18221e9200247aa5236993d"
server_key_iv: "14e0c55e7c245ad8c0d8e4e134e88e66"
server_key_version: "encryptionkey@0"
I20260619 02:57:07.645828 30153 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:07.652393 30153 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 02:57:07.652550 30153 log.cc:1196] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 02:57:07.653012 30153 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Deleting consensus metadata
I20260619 02:57:07.653263 30153 master_runner.cc:315] Copying system tablet from 127.22.230.62:38487
I20260619 02:57:07.666141 30153 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: tablet copy: Beginning tablet copy session from remote peer at address 127.22.230.62:38487
I20260619 02:57:07.688125 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.666915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42280)
0619 02:57:07.671010 (+ 4095us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:07.671030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:07.671171 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:07.671559 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:07.671564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:07.672061 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:07.672280 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.672292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.673791 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.673796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.675337 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:07.675351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.676359 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.676365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:07.676547 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.680224 (+ 3677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:07.680251 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:07.681307 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:07.685348 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.685353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.685358 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.685823 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.686200 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:07.686205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:07.686209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:07.686376 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:07.687111 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:07.687115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:07.687277 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 02:57:07.687410 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 02:57:07.687684 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3962,"thread_start_us":3818,"threads_started":1}
I20260619 02:57:07.688153 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:07.666931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.22.230.62:38487)
0619 02:57:07.667187 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:07.667193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:07.671133 (+ 3940us) server_negotiation.cc:407] Connection header received
0619 02:57:07.671298 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:07.671304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:07.671408 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:07.671515 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:07.672429 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.673624 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.675520 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:07.676187 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:07.676303 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:07.681496 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:07.681502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:07.681507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:07.681545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:07.685215 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.685946 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.685952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.685956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.686092 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:07.686494 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:07.686499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:07.686502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:07.686985 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:07.687140 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:07.687540 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 02:57:07.687782 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 02:57:07.689128 29612 tablet_copy_service.cc:140] P fe80c7302c424ac0bb2ce35787ef3f60: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0bb2a01f104647df976f364cdd0f0dd2 ({username='oryx', principal='oryx/127.22.230.60@KRBTEST.COM'} at 127.0.0.1:42280)
I20260619 02:57:07.689284 29612 tablet_copy_service.cc:161] P fe80c7302c424ac0bb2ce35787ef3f60: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0bb2a01f104647df976f364cdd0f0dd2 at {username='oryx', principal='oryx/127.22.230.60@KRBTEST.COM'} at 127.0.0.1:42280: session id = 0bb2a01f104647df976f364cdd0f0dd2-00000000000000000000000000000000
I20260619 02:57:07.692803 29612 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 02:57:07.694518 30153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 02:57:07.701494 30153 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: tablet copy: Starting download of 0 data blocks...
I20260619 02:57:07.701763 30153 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: tablet copy: Starting download of 1 WAL segments...
I20260619 02:57:07.703804 30153 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 02:57:07.708042 29612 tablet_copy_service.cc:342] P fe80c7302c424ac0bb2ce35787ef3f60: Request end of tablet copy session 0bb2a01f104647df976f364cdd0f0dd2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.230.60@KRBTEST.COM'} at 127.0.0.1:42280
I20260619 02:57:07.708166 29612 tablet_copy_service.cc:433] P fe80c7302c424ac0bb2ce35787ef3f60: ending tablet copy session 0bb2a01f104647df976f364cdd0f0dd2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0bb2a01f104647df976f364cdd0f0dd2
I20260619 02:57:07.718919 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:07.721576 30355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:07.724037 30356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:07.728260 30153 server_base.cc:1034] running on GCE node
W20260619 02:57:07.729282 30358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:07.730635 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:07.730703 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29653
I20260619 02:57:07.731505 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:07.732788 30153 hybrid_clock.cc:648] HybridClock initialized: now 1781837827732722 us; error 33 us; skew 500 ppm
I20260619 02:57:07.733007 30153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:07 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837827, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:07.735093 30153 init.cc:377] Logged in from keytab as oryx/127.22.230.60@KRBTEST.COM (short username oryx)
I20260619 02:57:07.739542 30153 webserver.cc:460] Webserver started at http://127.22.230.60:44797/ using document root <none> and password file <none>
I20260619 02:57:07.739897 30153 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:07.739985 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:07.742838 30153 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:57:07.744887 30365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:07.745246 30153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 02:57:07.745368 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
uuid: "0bb2a01f104647df976f364cdd0f0dd2"
format_stamp: "Formatted at 2026-06-19 02:57:07 on dist-test-slave-2gms"
server_key: "f7019779c18221e9200247aa5236993d"
server_key_iv: "14e0c55e7c245ad8c0d8e4e134e88e66"
server_key_version: "encryptionkey@0"
I20260619 02:57:07.745555 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:07.750411 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:34993
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:38487
--webserver_port=40859
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:57:07.773238 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:07.777259 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:07.814440 30153 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.60:38553
I20260619 02:57:07.814627 30417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.60:38553 every 8 connection(s)
I20260619 02:57:07.815485 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
I20260619 02:57:07.820740 30418 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:57:07.824487 30418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Bootstrap starting.
I20260619 02:57:07.838860 30418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:57:07.839332 30418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: Bootstrap complete.
I20260619 02:57:07.840332 30418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } }
I20260619 02:57:07.840634 30418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0bb2a01f104647df976f364cdd0f0dd2, State: Initialized, Role: LEARNER
I20260619 02:57:07.840870 30418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } }
I20260619 02:57:07.841686 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:07.842036 30418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:07.842061 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:57:07.864518 30432 catalog_manager.cc:1259] Loaded cluster ID: f220dbd2e48f42848627ebe0b42712d8
I20260619 02:57:07.864621 30432 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: loading cluster ID for follower catalog manager: success
I20260619 02:57:07.868232 30432 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: acquiring CA information for follower catalog manager: success
I20260619 02:57:07.870481 30432 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 02:57:08.101758 30370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:08.102000 30370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:08.102084 30370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:08.109042 30370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:08.109136 30370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:08.109237 30370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:08.120908 30370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:34993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=40859
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:08.121207 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.114090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43954 (local address 127.22.230.60:38553)
0619 02:57:08.114642 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.114648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.114877 (+ 229us) server_negotiation.cc:407] Connection header received
0619 02:57:08.115048 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.115053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.115141 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.115275 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:08.115605 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.117114 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.119036 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.120247 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.120374 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.120628 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.120767 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.120893 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
I20260619 02:57:08.121243 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.113939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.0.0.1:43954)
0619 02:57:08.114781 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.114796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.114907 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.115290 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.115293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.115311 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:08.115473 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.115479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.117258 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.117262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.118891 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.118902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.120392 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.120396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.120624 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.120648 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.120745 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.120882 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":631,"threads_started":1}
I20260619 02:57:08.122103 30393 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 02:57:08.122398 30370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:08.122479 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe80c7302c424ac0bb2ce35787ef3f60. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:08.122766 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: LEARNER
I20260619 02:57:08.123817 30370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:08.138581 30442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:08.138633 30440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:08.138622 30439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:08.139727 30370 server_base.cc:1034] running on GCE node
I20260619 02:57:08.140281 30370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:08.141181 30370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:08.142340 30370 hybrid_clock.cc:648] HybridClock initialized: now 1781837828142283 us; error 50 us; skew 500 ppm
I20260619 02:57:08.142660 30370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:08.145776 30370 init.cc:377] Logged in from keytab as oryx/127.22.230.1@KRBTEST.COM (short username oryx)
I20260619 02:57:08.147225 30370 webserver.cc:460] Webserver started at http://127.22.230.1:40859/ using document root <none> and password file <none>
I20260619 02:57:08.147863 30370 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:08.148005 30370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:08.153769 30370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:57:08.156586 30449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:08.157809 30370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:08.157963 30370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "c2d12dd94a5c44168357171a07737412"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "a1b31692e45f8d56032f9639bc840280"
server_key_iv: "6aae7ce6cc0c22793ac74f91a42174c1"
server_key_version: "encryptionkey@0"
I20260619 02:57:08.159000 30370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:08.187239 30370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:08.191045 30370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:08.191308 30370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:08.192570 30370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:08.193773 30370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:08.193848 30370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:08.193982 30370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:08.194072 30370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:08.228936 30370 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:34993
I20260619 02:57:08.229017 30561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:34993 every 8 connection(s)
I20260619 02:57:08.230304 30370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:08.233124 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 30370
I20260619 02:57:08.233435 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29787
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.2@KRBTEST.COM
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.60@KRBTEST.COM
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:08.264433 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.235264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:48637 (local address 127.22.230.60:38553)
0619 02:57:08.236297 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.236306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.247227 (+ 10921us) server_negotiation.cc:407] Connection header received
0619 02:57:08.247340 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.247345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.247415 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.247548 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.248713 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249696 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.251929 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.252425 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.252530 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.257096 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.257100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.257103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.257131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.260450 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.261346 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.261350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.261355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.261511 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.262107 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.262112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.262117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.262529 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.262681 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.263804 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.264039 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865}
I20260619 02:57:08.265355 30571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.235492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:35363 (local address 127.22.230.61:43577)
0619 02:57:08.236084 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.236093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.243882 (+ 7789us) server_negotiation.cc:407] Connection header received
0619 02:57:08.245491 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.245496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.245574 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.245660 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.247970 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249408 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.252256 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.252926 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.253039 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.258178 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.258185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.258190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.258231 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.262250 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.263083 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.263088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.263093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.263235 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.263639 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.263644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.263647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.263966 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.264092 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.264813 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.265003 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":834,"threads_started":1}
I20260619 02:57:08.265970 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.233779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.22.230.1:48637)
0619 02:57:08.247113 (+ 13334us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.247129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.247247 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.247719 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.247723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.248243 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:08.248546 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.248557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249863 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.249870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.251770 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.251787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.252862 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.252870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.253207 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.253868 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:08.253906 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:08.256897 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:08.260582 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.260592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.260609 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.261217 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.261663 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.261668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.261673 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.261895 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.262765 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:08.262772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:08.263261 (+ 489us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.264641 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.264892 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13180,"mutex_wait_us":1660,"thread_start_us":9527,"threads_started":1}
I20260619 02:57:08.266355 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.234510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:34787 (local address 127.22.230.62:38487)
0619 02:57:08.239935 (+ 5425us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.239944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.245143 (+ 5199us) server_negotiation.cc:407] Connection header received
0619 02:57:08.245317 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.245321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.245400 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.245496 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.247970 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249443 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.253458 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.254122 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.254269 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.258533 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.258541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.258564 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.258625 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.261895 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.262764 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.262771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.262776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.262907 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.263332 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.263337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.263342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.263688 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.263822 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.265822 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.266016 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5272,"thread_start_us":5173,"threads_started":1}
I20260619 02:57:08.266683 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.234554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:34787)
0619 02:57:08.245044 (+ 10490us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.245058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.245186 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.245935 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.245945 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.246789 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:08.247762 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.247787 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249612 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.249616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.253310 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.253325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.254500 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.254507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.254729 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.255406 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:08.255443 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:08.258319 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:08.262040 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.262044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.262048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.262620 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.263035 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.263042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.263049 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.263202 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.263876 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:08.263881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:08.264005 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.264151 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.264625 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10334,"mutex_wait_us":266,"thread_start_us":12222,"threads_started":1}
I20260619 02:57:08.267401 30566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.233777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.22.230.1:35363)
0619 02:57:08.240853 (+ 7076us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.240904 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.244446 (+ 3542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.245949 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.245952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.247404 (+ 1452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:08.247762 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.247787 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249633 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.249639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.252081 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.252099 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.253545 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.253554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.253738 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.254376 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:08.254397 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:08.257915 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:08.262481 (+ 4566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.262485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.262489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.262966 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.263364 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.263368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.263373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.263519 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.264173 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:08.264178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:08.264303 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.265728 (+ 1425us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.265994 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6813,"mutex_wait_us":119,"thread_start_us":6582,"threads_started":1}
I20260619 02:57:08.268184 30563 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38487
I20260619 02:57:08.268211 30562 heartbeater.cc:347] Connected to a master server at 127.22.230.60:38553
I20260619 02:57:08.268283 30564 heartbeater.cc:347] Connected to a master server at 127.22.230.61:43577
I20260619 02:57:08.268512 30562 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:08.268508 30564 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:08.268508 30563 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:08.269151 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.235907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.22.230.2:34311)
0619 02:57:08.240443 (+ 4536us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.240453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.240487 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:57:08.240572 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.240583 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.240870 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.241103 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.244156 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.246749 (+ 2593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.248759 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.249506 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.249611 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.258702 (+ 9091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.259431 (+ 729us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.259454 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.259507 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.263556 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.264480 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.264489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.264491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.264709 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.265110 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.265115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.265119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.265516 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.268084 (+ 2568us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.268250 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.268510 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4347,"thread_start_us":4185,"threads_started":1}
I20260619 02:57:08.269512 30563 heartbeater.cc:511] Master 127.22.230.62:38487 requested a full tablet report, sending...
I20260619 02:57:08.269534 30564 heartbeater.cc:511] Master 127.22.230.61:43577 requested a full tablet report, sending...
I20260619 02:57:08.269971 30562 heartbeater.cc:511] Master 127.22.230.60:38553 requested a full tablet report, sending...
I20260619 02:57:08.270288 29910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:08.271217 29592 ts_manager.cc:194] Re-registered known tserver with Master: c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993)
I20260619 02:57:08.271595 30190 ts_manager.cc:194] Registered new tserver with Master: c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993)
I20260619 02:57:08.271874 30383 ts_manager.cc:194] Registered new tserver with Master: c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993)
I20260619 02:57:08.272280 29592 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:34787
I20260619 02:57:08.353305 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29787
I20260619 02:57:08.370132 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:34311
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:38487
--webserver_port=46173
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:57:08.623417 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: NON_VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: true } } }
I20260619 02:57:08.623693 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:08.626783 30575 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: attempting to promote NON_VOTER 0bb2a01f104647df976f364cdd0f0dd2 to VOTER
I20260619 02:57:08.627287 30575 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:08.629091 30393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 LEARNER]: Refusing update from remote peer fe80c7302c424ac0bb2ce35787ef3f60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 02:57:08.629297 30200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Refusing update from remote peer fe80c7302c424ac0bb2ce35787ef3f60: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 02:57:08.629633 30576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 02:57:08.629995 30335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 02:57:08.632854 30576 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.633464 30393 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.637645 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.638003 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:08.638278 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.638581 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:08.642236 30581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.642632 30581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:57:08.643816 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.644074 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:08.644276 30200 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.647229 30582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:08.647590 30582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 02:57:08.687103 30574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:08.687345 30574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:08.687428 30574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:08.693964 30574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:08.694048 30574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:08.694147 30574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:08.703009 30574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:34311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=46173
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:08.704106 30574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:08.705265 30574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:08.717561 30587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:08.717599 30588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:08.717567 30590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:08.718779 30574 server_base.cc:1034] running on GCE node
I20260619 02:57:08.719326 30574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:08.720115 30574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:08.721323 30574 hybrid_clock.cc:648] HybridClock initialized: now 1781837828721291 us; error 47 us; skew 500 ppm
I20260619 02:57:08.721661 30574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:08.724453 30574 init.cc:377] Logged in from keytab as oryx/127.22.230.2@KRBTEST.COM (short username oryx)
I20260619 02:57:08.725719 30574 webserver.cc:460] Webserver started at http://127.22.230.2:46173/ using document root <none> and password file <none>
I20260619 02:57:08.726336 30574 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:08.726481 30574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:08.731309 30574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:57:08.733757 30597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:08.735024 30574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:57:08.735188 30574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "b4baead2a72f46ac962b668e105f2c7e"
format_stamp: "Formatted at 2026-06-19 02:57:05 on dist-test-slave-2gms"
server_key: "43b138e564260a57dc8a199d3e3059ec"
server_key_iv: "c551339a9d1284b6ab81cfe5ea4562cf"
server_key_version: "encryptionkey@0"
I20260619 02:57:08.736147 30574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:08.760851 30574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:08.764532 30574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:08.764793 30574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:08.765678 30574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:08.767076 30574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:08.767189 30574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:08.767304 30574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:08.767392 30574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:08.801725 30574 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:34311
I20260619 02:57:08.801833 30709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:34311 every 8 connection(s)
I20260619 02:57:08.803148 30574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:08.809300 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 30574
I20260619 02:57:08.809580 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29920
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837828, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.60@KRBTEST.COM
I20260619 02:57:08.833102 30714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.806741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:39249 (local address 127.22.230.62:38487)
0619 02:57:08.807247 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.807254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.811026 (+ 3772us) server_negotiation.cc:407] Connection header received
0619 02:57:08.811107 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.811115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.811214 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.811328 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.814406 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.815929 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.818870 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.819748 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.819874 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.825719 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.825727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.825732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.825782 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.829158 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.830136 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.830142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.830147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.830291 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.830830 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.830835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.831056 (+ 221us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.831398 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.831548 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.832496 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.832702 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":435,"threads_started":1}
I20260619 02:57:08.835186 30716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.806678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:39249)
0619 02:57:08.807811 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.807862 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.810656 (+ 2794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.811546 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.811553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.812642 (+ 1089us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:08.814205 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.814236 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.816170 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.816179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.818647 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.818666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.820239 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.820247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.820483 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.821216 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:08.821255 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:08.825450 (+ 4195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:08.829306 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.829315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.829330 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.829926 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.830430 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.830434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.830437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.830688 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.831644 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:08.831653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:08.832074 (+ 421us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.833299 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.833811 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":199,"spinlock_wait_cycles":8960,"thread_start_us":3390,"threads_started":1}
Jun 19 02:57:08 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.3@KRBTEST.COM
I20260619 02:57:08.835945 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.806995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.22.230.2:47957)
0619 02:57:08.808656 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.808679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.810746 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.811530 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.811541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.813060 (+ 1519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:08.814742 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.814757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.816247 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.816252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.818510 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.818523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.819688 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.819694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.819978 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.821441 (+ 1463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:08.821461 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:08.826005 (+ 4544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:08.830006 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.830013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.830019 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.830579 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.833794 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.833800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.833804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.833975 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.834634 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:08.834641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:08.834783 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.834917 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.835171 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":157,"thread_start_us":344,"threads_started":1}
I20260619 02:57:08.836774 30711 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38487
I20260619 02:57:08.837105 30711 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:08.837376 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.807913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:47957 (local address 127.22.230.61:43577)
0619 02:57:08.809072 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.809076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.809091 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:57:08.811051 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.811057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.811137 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.812403 (+ 1266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.814899 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.816087 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.818717 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.819330 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.819439 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.826199 (+ 6760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.826205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.826210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.826250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.829853 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.833497 (+ 3644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.833504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.833509 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.833649 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.834104 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.834108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.834111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.834421 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.834556 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.836442 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.836646 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":1614,"threads_started":1}
I20260619 02:57:08.837976 30711 heartbeater.cc:511] Master 127.22.230.62:38487 requested a full tablet report, sending...
I20260619 02:57:08.838136 30715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.807562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.22.230.2:60499)
0619 02:57:08.808682 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:08.808696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:08.810659 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:08.811673 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:08.811679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:08.812657 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:08.814235 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.814244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.823366 (+ 9122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.823371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.825299 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:08.825312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.827826 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.827833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:08.828035 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.828516 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:08.828538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:08.831602 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:08.835252 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.835256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.835259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.835763 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.836188 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:08.836192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:08.836196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:08.836342 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:08.836986 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:08.836991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:08.837266 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 02:57:08.837400 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 02:57:08.837665 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":205,"spinlock_wait_cycles":25472,"thread_start_us":2460,"threads_started":1}
I20260619 02:57:08.839179 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.808705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:60499 (local address 127.22.230.60:38553)
0619 02:57:08.809317 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.809321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.809334 (+ 13us) server_negotiation.cc:407] Connection header received
0619 02:57:08.811453 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.811458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.811532 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.811825 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.816920 (+ 5095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.823074 (+ 6154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.826994 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.827637 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.828317 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.831881 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.831887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.831892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.831921 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.835111 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.835899 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.835904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.835907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.836031 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.836473 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.836478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.836482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.836828 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.838354 (+ 1526us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.838472 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.838717 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":2067,"threads_started":1}
I20260619 02:57:08.839205 30712 heartbeater.cc:347] Connected to a master server at 127.22.230.61:43577
I20260619 02:57:08.839318 30712 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:08.839740 30712 heartbeater.cc:511] Master 127.22.230.61:43577 requested a full tablet report, sending...
I20260619 02:57:08.839890 30710 heartbeater.cc:347] Connected to a master server at 127.22.230.60:38553
I20260619 02:57:08.840004 30710 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:08.840409 30710 heartbeater.cc:511] Master 127.22.230.60:38553 requested a full tablet report, sending...
I20260619 02:57:08.840965 29592 ts_manager.cc:194] Re-registered known tserver with Master: b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:08.841140 30190 ts_manager.cc:194] Registered new tserver with Master: b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:08.841452 30383 ts_manager.cc:194] Registered new tserver with Master: b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:08.842152 29592 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:39249
I20260619 02:57:08.847306 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:08.810593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.22.230.3:40529)
0619 02:57:08.823408 (+ 12815us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:08.823417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:08.823445 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:57:08.823521 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:08.823531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:08.823786 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:08.823995 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:08.826246 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.827963 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.829776 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:08.830728 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:08.830843 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:08.837778 (+ 6935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:08.837791 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:08.837806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:08.837851 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:08.844584 (+ 6733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.845389 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.845395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.845397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.845541 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:08.845883 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:08.845886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:08.845888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:08.846182 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:08.846436 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:08.846751 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 02:57:08.846921 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12557,"thread_start_us":12234,"threads_started":1}
I20260619 02:57:08.848421 30043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:08.928488 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29920
I20260619 02:57:08.946039 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:40529
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:38487
--webserver_port=43117
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:09.259094 30723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:09.259356 30723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:09.259529 30723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:09.266971 30723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:09.267117 30723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:09.267298 30723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:09.274297 30563 heartbeater.cc:503] Master 127.22.230.62:38487 was elected leader, sending a full tablet report...
I20260619 02:57:09.277801 30723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:40529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=43117
--tserver_master_addrs=127.22.230.62:38487,127.22.230.61:43577,127.22.230.60:38553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:09.279124 30723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:09.280391 30723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:09.292618 30729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:09.292644 30728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:09.292814 30731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:09.293498 30723 server_base.cc:1034] running on GCE node
I20260619 02:57:09.294098 30723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:09.295049 30723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:09.296262 30723 hybrid_clock.cc:648] HybridClock initialized: now 1781837829296210 us; error 44 us; skew 500 ppm
I20260619 02:57:09.296545 30723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:09 dist-test-slave-2gms krb5kdc[29547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837829, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:09.299461 30723 init.cc:377] Logged in from keytab as oryx/127.22.230.3@KRBTEST.COM (short username oryx)
I20260619 02:57:09.300670 30723 webserver.cc:460] Webserver started at http://127.22.230.3:43117/ using document root <none> and password file <none>
I20260619 02:57:09.301256 30723 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:09.301357 30723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:09.306232 30723 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:57:09.308797 30738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:09.310037 30723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:57:09.310199 30723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "48bad7dcc83b4d3d9f455747f7b85970"
format_stamp: "Formatted at 2026-06-19 02:57:06 on dist-test-slave-2gms"
server_key: "2211a4d76494cd807084288991d6a5b3"
server_key_iv: "d6a8acd570d977870382262b8ddc9b5d"
server_key_version: "encryptionkey@0"
I20260619 02:57:09.311189 30723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:09.333438 30723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:09.337464 30723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:09.337733 30723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:09.338701 30723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:09.340126 30723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:09.340232 30723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:09.340343 30723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:09.340432 30723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:09.372251 30723 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40529
I20260619 02:57:09.372334 30850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40529 every 8 connection(s)
I20260619 02:57:09.373562 30723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:09.376632 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 30723
Jun 19 02:57:09 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837829, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.61@KRBTEST.COM
I20260619 02:57:09.426105 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.392902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.22.230.61:43577)
0619 02:57:09.403323 (+ 10421us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.403335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.403361 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:57:09.403441 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.403449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.403546 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.407116 (+ 3570us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.407308 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.408847 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.412336 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.412772 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.412891 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.416922 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.416932 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.416940 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.416997 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.420810 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.422758 (+ 1948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.422763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.422767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.422913 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.423299 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.423304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.423307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.423901 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.424159 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.425369 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.425608 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":9513,"thread_start_us":9404,"threads_started":1}
Jun 19 02:57:09 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837829, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:09.430330 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.378605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:56903 (local address 127.22.230.61:43577)
0619 02:57:09.389647 (+ 11042us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.389651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.406830 (+ 17179us) server_negotiation.cc:407] Connection header received
0619 02:57:09.407102 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.407110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.407228 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.407312 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.410348 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.411786 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.415017 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.415735 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.415823 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.422169 (+ 6346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.422177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.422181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.422226 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.426434 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.427849 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.427854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.427858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.427976 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.428515 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.428523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.428526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.428815 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.428942 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.429899 (+ 957us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.430062 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":10931,"spinlock_wait_cycles":8960,"thread_start_us":10813,"threads_started":1}
I20260619 02:57:09.432626 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.376871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.22.230.3:56903)
0619 02:57:09.406712 (+ 29841us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.406733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.406928 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.407957 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.407972 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.409001 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:09.409930 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.409968 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.412005 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.412012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.414845 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.414874 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.416666 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.416673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.416934 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.417587 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:09.417622 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:09.421925 (+ 4303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:09.427132 (+ 5207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:09.427140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:09.427154 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:09.427725 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:09.428087 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:09.428092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:09.428096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:09.428387 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:09.429078 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:09.429086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:09.429471 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.430612 (+ 1141us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.431170 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29655,"mutex_wait_us":344,"thread_start_us":28172,"threads_started":1}
Jun 19 02:57:09 dist-test-slave-2gms krb5kdc[29547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837829, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.60@KRBTEST.COM
I20260619 02:57:09.434994 30853 heartbeater.cc:347] Connected to a master server at 127.22.230.61:43577
I20260619 02:57:09.435333 30853 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:09.436240 30853 heartbeater.cc:511] Master 127.22.230.61:43577 requested a full tablet report, sending...
I20260619 02:57:09.437245 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.378905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:46417 (local address 127.22.230.62:38487)
0619 02:57:09.388975 (+ 10070us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.388984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.415596 (+ 26612us) server_negotiation.cc:407] Connection header received
0619 02:57:09.419743 (+ 4147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.419750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.419831 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.419928 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.421197 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.421979 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.423764 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.424407 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.424498 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.429853 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.429860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.429864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.429900 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.434256 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.435091 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.435096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.435099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.435232 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.435711 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.435715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.435718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.436041 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.436166 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.436681 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.436884 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":9904,"thread_start_us":9808,"threads_started":1}
I20260619 02:57:09.437878 30190 ts_manager.cc:194] Registered new tserver with Master: 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529)
I20260619 02:57:09.438112 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.377555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:46417)
0619 02:57:09.415484 (+ 37929us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.415501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.419593 (+ 4092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.420088 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.420094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.420580 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:09.421054 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.421070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.422102 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.422148 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.423617 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.423630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.424745 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.424751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.424923 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.425455 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:09.425474 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:09.429656 (+ 4182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:09.434396 (+ 4740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:09.434400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:09.434406 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:09.434953 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:09.435375 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:09.435380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:09.435383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:09.435556 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:09.436260 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:09.436265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:09.436396 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.437479 (+ 1083us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.437740 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37825,"thread_start_us":38056,"threads_started":1}
I20260619 02:57:09.438954 30852 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38487
I20260619 02:57:09.439080 30852 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:09.439931 30852 heartbeater.cc:511] Master 127.22.230.62:38487 requested a full tablet report, sending...
I20260619 02:57:09.440912 29592 ts_manager.cc:194] Re-registered known tserver with Master: 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529)
I20260619 02:57:09.442045 29592 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:46417
I20260619 02:57:09.444378 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.376871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.22.230.3:37049)
0619 02:57:09.402025 (+ 25154us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.402094 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.405000 (+ 2906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.413016 (+ 8016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.413021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.413588 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:09.413835 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.413847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.417703 (+ 3856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.417707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.419399 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.419419 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.430245 (+ 10826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.430253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.430436 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.431944 (+ 1508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:09.431964 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:09.435409 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:09.441399 (+ 5990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:09.441403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:09.441407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:09.441856 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:09.442336 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:09.442343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:09.442347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:09.442538 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:09.443250 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:09.443258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:09.443431 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.443568 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.443876 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24772,"thread_start_us":33114,"threads_started":1}
I20260619 02:57:09.444641 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.395046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.22.230.60:38553)
0619 02:57:09.419974 (+ 24928us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.419982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.419998 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:09.420050 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.420055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.420122 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.422822 (+ 2700us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.423020 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.424754 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.429264 (+ 4510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.429916 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.435609 (+ 5693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.435864 (+ 255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.435871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.435874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.435907 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.440041 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.441893 (+ 1852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.441898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.441901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.442062 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.442485 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.442489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.442492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.442907 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.443144 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.444051 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.444239 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24820,"thread_start_us":18885,"threads_started":1}
I20260619 02:57:09.445351 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.377170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:37049 (local address 127.22.230.60:38553)
0619 02:57:09.412591 (+ 35421us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.412597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.412615 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:57:09.412668 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.412673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.412750 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.412853 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.415845 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.417513 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.426645 (+ 9132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.427281 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.430747 (+ 3466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.435593 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.435599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.435603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.435637 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.441263 (+ 5626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.442023 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.442028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.442032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.442183 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.442737 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.442741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.442744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.443069 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.444583 (+ 1514us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.444728 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.444942 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35310,"spinlock_wait_cycles":1920,"thread_start_us":50193,"threads_started":1}
I20260619 02:57:09.446084 30851 heartbeater.cc:347] Connected to a master server at 127.22.230.60:38553
I20260619 02:57:09.446205 30851 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:09.446514 30851 heartbeater.cc:511] Master 127.22.230.60:38553 requested a full tablet report, sending...
I20260619 02:57:09.447564 30383 ts_manager.cc:194] Registered new tserver with Master: 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529)
I20260619 02:57:09.448683 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.395672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.22.230.62:38487)
0619 02:57:09.411878 (+ 16206us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.411884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.420702 (+ 8818us) server_negotiation.cc:407] Connection header received
0619 02:57:09.420895 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.420902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.420971 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.421101 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.426965 (+ 5864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.428099 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.430660 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.431712 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.431854 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.441357 (+ 9503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.441370 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.441376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.441412 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.446065 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.446736 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.446741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.446744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.446877 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.447296 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.447300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.447302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.447525 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.447779 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.448096 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.448334 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16086,"thread_start_us":16212,"threads_started":1}
I20260619 02:57:09.451978 29592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:57:09.459365 29592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:09.491463 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.476314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:34993 (local address 127.0.0.1:48014)
0619 02:57:09.477205 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.477224 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.477341 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.481211 (+ 3870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.481219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.481253 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:09.481766 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.481780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.485762 (+ 3982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.485771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.488575 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.488592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.490210 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.490216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.490402 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.490438 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.490586 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.490705 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"threads_started":1}
I20260619 02:57:09.491472 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.478353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.22.230.1:34993)
0619 02:57:09.480399 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.480412 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.480448 (+ 36us) server_negotiation.cc:407] Connection header received
0619 02:57:09.480563 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.480575 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.480927 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.482015 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:09.482271 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.485572 (+ 3301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.488766 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.489996 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.490163 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.490377 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.490628 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.490771 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":370,"threads_started":1}
I20260619 02:57:09.499361 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.476667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40529 (local address 127.0.0.1:43162)
0619 02:57:09.478160 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.478182 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.478312 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.490218 (+ 11906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.490225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.490258 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:09.490889 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.490904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.494018 (+ 3114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.494022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.496256 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.496268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.498615 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.498623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.498788 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.498819 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.498911 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.499025 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":617,"threads_started":1}
I20260619 02:57:09.499432 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.477140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.22.230.3:40529)
0619 02:57:09.488467 (+ 11327us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.488479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.488517 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:57:09.488619 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.488629 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.488907 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.491126 (+ 2219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:09.491442 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.493883 (+ 2441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.496913 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.498362 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.498509 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.498730 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.498958 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.499111 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11139,"thread_start_us":10968,"threads_started":1}
I20260619 02:57:09.499150 30497 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9d728616184f2e81d07b27ed15b8f1 (DEFAULT_TABLE table=test-table [id=6446ec81b0874d1080cc844a57c4be8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:09.500689 30497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9d728616184f2e81d07b27ed15b8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:09.500844 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.477312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:34311 (local address 127.0.0.1:52634)
0619 02:57:09.491673 (+ 14361us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.491689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.491770 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.494553 (+ 2783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.494559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.494638 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:09.494789 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.494815 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.496609 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.496614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.498436 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.498451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.500170 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.500175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.500359 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.500388 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.500470 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.500582 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14261,"thread_start_us":1359,"threads_started":1}
I20260619 02:57:09.501446 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.477921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.22.230.2:34311)
0619 02:57:09.493584 (+ 15663us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.493591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.493617 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:57:09.493708 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.493715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.493927 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.494136 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:09.494992 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.496461 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.498653 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.499955 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.500077 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.500263 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.501049 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.501212 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15517,"thread_start_us":16415,"threads_started":1}
I20260619 02:57:09.502524 30786 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9d728616184f2e81d07b27ed15b8f1 (DEFAULT_TABLE table=test-table [id=6446ec81b0874d1080cc844a57c4be8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:09.504289 30786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9d728616184f2e81d07b27ed15b8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:09.506139 30645 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9d728616184f2e81d07b27ed15b8f1 (DEFAULT_TABLE table=test-table [id=6446ec81b0874d1080cc844a57c4be8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:09.507378 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9d728616184f2e81d07b27ed15b8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:09.519227 30885 tablet_bootstrap.cc:492] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Bootstrap starting.
I20260619 02:57:09.521000 30886 tablet_bootstrap.cc:492] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Bootstrap starting.
I20260619 02:57:09.523129 30885 tablet_bootstrap.cc:654] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:09.523756 30887 tablet_bootstrap.cc:492] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Bootstrap starting.
I20260619 02:57:09.524427 30885 log.cc:824] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:09.525086 30886 tablet_bootstrap.cc:654] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:09.526117 30886 log.cc:824] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:09.527696 30887 tablet_bootstrap.cc:654] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:09.528008 30885 tablet_bootstrap.cc:492] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: No bootstrap required, opened a new log
I20260619 02:57:09.528295 30885 ts_tablet_manager.cc:1397] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 02:57:09.528816 30887 log.cc:824] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:09.529848 30886 tablet_bootstrap.cc:492] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: No bootstrap required, opened a new log
I20260619 02:57:09.530129 30886 ts_tablet_manager.cc:1397] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 02:57:09.531522 30887 tablet_bootstrap.cc:492] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: No bootstrap required, opened a new log
I20260619 02:57:09.531766 30887 ts_tablet_manager.cc:1397] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:57:09.536672 30885 raft_consensus.cc:348] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.537693 30885 raft_consensus.cc:374] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:09.537820 30885 raft_consensus.cc:729] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d12dd94a5c44168357171a07737412, State: Initialized, Role: FOLLOWER
I20260619 02:57:09.538399 30885 consensus_queue.cc:260] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.540578 30886 raft_consensus.cc:348] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.540987 30886 raft_consensus.cc:374] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:09.541087 30886 raft_consensus.cc:729] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bad7dcc83b4d3d9f455747f7b85970, State: Initialized, Role: FOLLOWER
I20260619 02:57:09.541627 30886 consensus_queue.cc:260] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.541829 30885 ts_tablet_manager.cc:1428] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260619 02:57:09.546345 30891 raft_consensus.cc:482] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:09.546732 30891 raft_consensus.cc:504] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.551031 30887 raft_consensus.cc:348] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.552361 30887 raft_consensus.cc:374] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:09.552496 30887 raft_consensus.cc:729] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4baead2a72f46ac962b668e105f2c7e, State: Initialized, Role: FOLLOWER
I20260619 02:57:09.553354 30887 consensus_queue.cc:260] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.555146 30891 leader_election.cc:288] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529), b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:09.555811 30886 ts_tablet_manager.cc:1428] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Time spent starting tablet: real 0.026s user 0.015s sys 0.012s
I20260619 02:57:09.555848 30852 heartbeater.cc:503] Master 127.22.230.62:38487 was elected leader, sending a full tablet report...
I20260619 02:57:09.564021 30887 ts_tablet_manager.cc:1428] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Time spent starting tablet: real 0.032s user 0.014s sys 0.006s
I20260619 02:57:09.564322 30711 heartbeater.cc:503] Master 127.22.230.62:38487 was elected leader, sending a full tablet report...
I20260619 02:57:09.564666 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.548794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:39123 (local address 127.22.230.3:40529)
0619 02:57:09.549037 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.549046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.549782 (+ 736us) server_negotiation.cc:407] Connection header received
0619 02:57:09.550091 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.550099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.550195 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.550271 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:09.551235 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.552847 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.562489 (+ 9642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.563682 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.563841 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.563902 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.564298 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.564430 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 02:57:09.564848 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.549904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:43053 (local address 127.22.230.2:34311)
0619 02:57:09.550119 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.550127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.556206 (+ 6079us) server_negotiation.cc:407] Connection header received
0619 02:57:09.556711 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.556717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.556787 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.556851 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:09.557587 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.558616 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.560899 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.562721 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.562872 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.562959 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.564478 (+ 1519us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.564605 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:57:09.565380 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.548671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40529 (local address 127.22.230.1:39123)
0619 02:57:09.549656 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.549680 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.549909 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.550539 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.550547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.550600 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:09.551043 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.551058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.553130 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.553137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.562258 (+ 9121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.562271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.563897 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.563903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.564101 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.564141 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.564811 (+ 670us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.564970 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":603,"threads_started":1}
I20260619 02:57:09.565928 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.549840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:34311 (local address 127.22.230.1:43053)
0619 02:57:09.556102 (+ 6262us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:09.556122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:09.556430 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:09.557066 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:09.557072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:09.557095 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:09.557368 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.557376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.558779 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.558784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.560732 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:09.560748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.564092 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.564098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:09.564290 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.564330 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 02:57:09.565064 (+ 734us) client_negotiation.cc:240] Negotiation successful
0619 02:57:09.565203 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5120,"thread_start_us":4977,"threads_started":1}
I20260619 02:57:09.566692 30806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9d728616184f2e81d07b27ed15b8f1" candidate_uuid: "c2d12dd94a5c44168357171a07737412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48bad7dcc83b4d3d9f455747f7b85970" is_pre_election: true
I20260619 02:57:09.567195 30806 raft_consensus.cc:2454] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d12dd94a5c44168357171a07737412 in term 0.
I20260619 02:57:09.567353 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9d728616184f2e81d07b27ed15b8f1" candidate_uuid: "c2d12dd94a5c44168357171a07737412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4baead2a72f46ac962b668e105f2c7e" is_pre_election: true
I20260619 02:57:09.567831 30665 raft_consensus.cc:2454] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d12dd94a5c44168357171a07737412 in term 0.
I20260619 02:57:09.567857 30451 leader_election.cc:302] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48bad7dcc83b4d3d9f455747f7b85970, c2d12dd94a5c44168357171a07737412; no voters:
I20260619 02:57:09.568295 30891 raft_consensus.cc:2781] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:09.568440 30891 raft_consensus.cc:482] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:09.568555 30891 raft_consensus.cc:3037] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:09.571254 30891 raft_consensus.cc:504] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.571831 30891 leader_election.cc:288] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [CANDIDATE]: Term 1 election: Requested vote from peers 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529), b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:09.572265 30806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9d728616184f2e81d07b27ed15b8f1" candidate_uuid: "c2d12dd94a5c44168357171a07737412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48bad7dcc83b4d3d9f455747f7b85970"
I20260619 02:57:09.572325 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9d728616184f2e81d07b27ed15b8f1" candidate_uuid: "c2d12dd94a5c44168357171a07737412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4baead2a72f46ac962b668e105f2c7e"
I20260619 02:57:09.572491 30665 raft_consensus.cc:3037] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:09.572503 30806 raft_consensus.cc:3037] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:09.575743 30806 raft_consensus.cc:2454] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d12dd94a5c44168357171a07737412 in term 1.
I20260619 02:57:09.575743 30665 raft_consensus.cc:2454] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d12dd94a5c44168357171a07737412 in term 1.
I20260619 02:57:09.576083 30451 leader_election.cc:302] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48bad7dcc83b4d3d9f455747f7b85970, c2d12dd94a5c44168357171a07737412; no voters:
I20260619 02:57:09.576391 30891 raft_consensus.cc:2781] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:09.576800 30891 raft_consensus.cc:686] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 1 LEADER]: Becoming Leader. State: Replica: c2d12dd94a5c44168357171a07737412, State: Running, Role: LEADER
I20260619 02:57:09.577283 30891 consensus_queue.cc:237] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:09.582504 29592 catalog_manager.cc:5495] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d12dd94a5c44168357171a07737412 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "c2d12dd94a5c44168357171a07737412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } health_report { overall_health: UNKNOWN } } }
I20260619 02:57:09.606472 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.601650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.22.230.1:34993)
0619 02:57:09.601817 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.601821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.601836 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:57:09.601913 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.601916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.601983 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.602103 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:09.602369 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.603147 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.604396 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.604952 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.605080 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.605477 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:09.605945 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:09.606056 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.606133 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.606190 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:57:09.614643 30806 raft_consensus.cc:1261] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Refusing update from remote peer c2d12dd94a5c44168357171a07737412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:09.614645 30665 raft_consensus.cc:1261] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Refusing update from remote peer c2d12dd94a5c44168357171a07737412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:09.615710 30896 consensus_queue.cc:1035] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:09.616050 30891 consensus_queue.cc:1035] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:09.628151 30899 mvcc.cc:205] Tried to move back new op lower bound from 7298407750091411456 to 7298407749952000000. Current Snapshot: MvccSnapshot[applied={T|T < 7298407750091411456}]
I20260619 02:57:09.632686 30900 mvcc.cc:205] Tried to move back new op lower bound from 7298407750091411456 to 7298407749952000000. Current Snapshot: MvccSnapshot[applied={T|T < 7298407750091411456}]
I20260619 02:57:09.640161 30901 mvcc.cc:205] Tried to move back new op lower bound from 7298407750091411456 to 7298407749952000000. Current Snapshot: MvccSnapshot[applied={T|T < 7298407750091411456}]
I20260619 02:57:09.652459 29592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:09.653179 29592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:09.659057 29592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe80c7302c424ac0bb2ce35787ef3f60: Sending DeleteTablet for 3 replicas of tablet 3b9d728616184f2e81d07b27ed15b8f1
I20260619 02:57:09.660212 30645 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9d728616184f2e81d07b27ed15b8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:09 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:52634
I20260619 02:57:09.660413 30497 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9d728616184f2e81d07b27ed15b8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:09 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:48014
I20260619 02:57:09.660941 30786 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9d728616184f2e81d07b27ed15b8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:09 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:43162
I20260619 02:57:09.669795 30911 tablet_replica.cc:330] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: stopping tablet replica
I20260619 02:57:09.670306 30911 raft_consensus.cc:2229] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:09.670876 30911 raft_consensus.cc:2258] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:09.673246 30909 tablet_replica.cc:330] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: stopping tablet replica
I20260619 02:57:09.673650 30909 raft_consensus.cc:2229] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:09.674098 30909 raft_consensus.cc:2258] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:09.675048 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 29560
I20260619 02:57:09.675191 30911 ts_tablet_manager.cc:1905] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:09.676820 29622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:09.678030 30909 ts_tablet_manager.cc:1905] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:09.682891 30911 ts_tablet_manager.cc:1918] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:09.683084 30911 log.cc:1196] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/3b9d728616184f2e81d07b27ed15b8f1
I20260619 02:57:09.683779 30911 ts_tablet_manager.cc:1939] T 3b9d728616184f2e81d07b27ed15b8f1 P 48bad7dcc83b4d3d9f455747f7b85970: Deleting consensus metadata
I20260619 02:57:09.685252 30910 tablet_replica.cc:330] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: stopping tablet replica
I20260619 02:57:09.685673 30910 raft_consensus.cc:2229] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:09.686424 30910 raft_consensus.cc:2258] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:09.688860 30910 ts_tablet_manager.cc:1905] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:09.689857 30909 ts_tablet_manager.cc:1918] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:09.690006 30909 log.cc:1196] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/3b9d728616184f2e81d07b27ed15b8f1
I20260619 02:57:09.690510 30909 ts_tablet_manager.cc:1939] T 3b9d728616184f2e81d07b27ed15b8f1 P b4baead2a72f46ac962b668e105f2c7e: Deleting consensus metadata
I20260619 02:57:09.694551 30910 ts_tablet_manager.cc:1918] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:09.694818 30910 log.cc:1196] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/3b9d728616184f2e81d07b27ed15b8f1
I20260619 02:57:09.695473 30910 ts_tablet_manager.cc:1939] T 3b9d728616184f2e81d07b27ed15b8f1 P c2d12dd94a5c44168357171a07737412: Deleting consensus metadata
I20260619 02:57:09.700906 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.663831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.22.230.61:43577)
0619 02:57:09.664047 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.664055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.676187 (+ 12132us) server_negotiation.cc:407] Connection header received
0619 02:57:09.676250 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.676255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.676329 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.676433 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.687664 (+ 11231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.688709 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.690625 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.691245 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.691374 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.694748 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.694755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.694761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.694802 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.697934 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.698651 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.698656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.698660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.698900 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.699436 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.699442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.699445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.699754 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.700008 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.700361 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.700582 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:57:09.713212 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.674429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.22.230.60:38553)
0619 02:57:09.674619 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.674626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.700928 (+ 26302us) server_negotiation.cc:407] Connection header received
0619 02:57:09.701512 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.701519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.701598 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.701666 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.703185 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.703895 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.705104 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.705600 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.705744 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.708166 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.708170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.708174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.708202 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.710809 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.711395 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.711407 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.711410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.711528 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.711881 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.711886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.711889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.712138 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.712387 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.712683 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.712888 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 02:57:09.869680 29577 catalog_manager.cc:4841] TS 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529): tablet 3b9d728616184f2e81d07b27ed15b8f1 (table test-table [id=6446ec81b0874d1080cc844a57c4be8c]) successfully deleted
W20260619 02:57:09.869967 29577 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.22.230.3:40529, user_credentials={real_user=slave}} blocked reactor thread for 184323us
I20260619 02:57:09.870280 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 29560
I20260619 02:57:09.921149 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.896649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.22.230.61:43577)
0619 02:57:09.896824 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.896831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.904370 (+ 7539us) server_negotiation.cc:407] Connection header received
0619 02:57:09.904709 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.904717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.904822 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.904928 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.908118 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.909520 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.910997 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.911424 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.911563 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.914766 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.914770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.914774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.914804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.918211 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.918993 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.919013 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.919019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.919138 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.919584 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.919589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.919593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.919975 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.920241 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.920575 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.920788 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 02:57:09.924016 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:09.896167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.22.230.60:38553)
0619 02:57:09.896363 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:09.896372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:09.908911 (+ 12539us) server_negotiation.cc:407] Connection header received
0619 02:57:09.908979 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:09.908984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:09.909059 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:09.909147 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:09.911634 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.912732 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.914725 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:09.915361 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:09.915474 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:09.918451 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:09.918461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:09.918469 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:09.918498 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:09.921532 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.922132 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.922137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.922141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.922271 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:09.922656 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:09.922660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:09.922664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:09.922926 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:09.923174 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:09.923472 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 02:57:09.923669 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:57:10.641542 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:10.640755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:50135)
0619 02:57:10.641233 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:10.641337 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":269,"threads_started":1}
W20260619 02:57:10.642036 30563 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:38487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:10.647058 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:10.646271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:34077)
0619 02:57:10.646721 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:10.646856 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":189,"threads_started":1}
W20260619 02:57:10.647469 30852 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:38487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:10.664889 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:10.664182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:34391)
0619 02:57:10.664625 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:10.664744 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":252,"threads_started":1}
W20260619 02:57:10.665274 30711 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:38487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:11.159682 30924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe80c7302c424ac0bb2ce35787ef3f60)
I20260619 02:57:11.159942 30924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:11.160306 30925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe80c7302c424ac0bb2ce35787ef3f60)
I20260619 02:57:11.160555 30925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:11.161511 30924 leader_election.cc:288] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487), 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61:43577)
I20260619 02:57:11.162863 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.161545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42478)
0619 02:57:11.162341 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.162524 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":530,"threads_started":1}
W20260619 02:57:11.163375 30366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:11.169294 30925 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487), 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60:38553)
I20260619 02:57:11.172060 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.161723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42482)
0619 02:57:11.171559 (+ 9836us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.171778 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8826,"threads_started":1}
W20260619 02:57:11.175515 30174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:11.176509 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.175373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42488)
0619 02:57:11.176095 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.176297 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574}
W20260619 02:57:11.176733 30366 leader_election.cc:334] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:11.178396 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.177886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42504)
0619 02:57:11.178083 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.178219 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 02:57:11.178601 30174 leader_election.cc:334] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:11.181210 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.162341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.22.230.61:43577)
0619 02:57:11.175081 (+ 12740us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:11.175087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:11.175104 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:57:11.175163 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:11.175167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:11.175249 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:11.175870 (+ 621us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:11.176056 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.177298 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.179412 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.180252 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.180405 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:11.180492 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:11.180776 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 02:57:11.180891 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12612,"thread_start_us":11070,"threads_started":1}
I20260619 02:57:11.181241 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.161723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43577 (local address 127.0.0.1:53670)
0619 02:57:11.169387 (+ 7664us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.169404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:11.169506 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:11.175409 (+ 5903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:11.175414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:11.175438 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:11.175727 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:11.175740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.177473 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.177481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:11.179269 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:11.179280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.180430 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.180436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:11.180608 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:11.180629 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:57:11.180759 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 02:57:11.180890 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7574,"thread_start_us":13303,"threads_started":1}
I20260619 02:57:11.181847 30200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bb2a01f104647df976f364cdd0f0dd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" is_pre_election: true
I20260619 02:57:11.182199 30932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.175306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.22.230.60:38553)
0619 02:57:11.176409 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:11.176413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:11.176429 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:11.176483 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:11.176487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:11.176561 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:11.176667 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:11.177522 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.178771 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.180314 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.181089 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.181225 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:11.181432 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:11.181721 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:57:11.181850 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1012,"thread_start_us":871,"threads_started":1}
I20260619 02:57:11.182246 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.169453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:38553 (local address 127.0.0.1:43990)
0619 02:57:11.172253 (+ 2800us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.172275 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:11.172411 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:11.176992 (+ 4581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:11.177002 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:11.177033 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:11.177357 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:11.177369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.178904 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.178911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:11.180172 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:11.180183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:11.181274 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:11.181280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:11.181571 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:11.181592 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 02:57:11.181691 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:57:11.181819 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2704,"mutex_wait_us":2,"thread_start_us":948,"threads_started":1}
I20260619 02:57:11.182271 30200 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb2a01f104647df976f364cdd0f0dd2 in term 1.
I20260619 02:57:11.182749 30393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0bb2a01f104647df976f364cdd0f0dd2" is_pre_election: true
I20260619 02:57:11.182835 30367 leader_election.cc:302] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bb2a01f104647df976f364cdd0f0dd2, 7ab12b4d6fb741a0aab3f5ee7bda9689; no voters: fe80c7302c424ac0bb2ce35787ef3f60
I20260619 02:57:11.183072 30393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ab12b4d6fb741a0aab3f5ee7bda9689 in term 1.
I20260619 02:57:11.183244 30924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:57:11.183359 30924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe80c7302c424ac0bb2ce35787ef3f60)
I20260619 02:57:11.183465 30924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:57:11.183601 30177 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bb2a01f104647df976f364cdd0f0dd2, 7ab12b4d6fb741a0aab3f5ee7bda9689; no voters: fe80c7302c424ac0bb2ce35787ef3f60
I20260619 02:57:11.183907 30925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 02:57:11.184020 30925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe80c7302c424ac0bb2ce35787ef3f60)
I20260619 02:57:11.184114 30925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:57:11.185621 30924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:11.186231 30924 leader_election.cc:288] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [CANDIDATE]: Term 2 election: Requested vote from peers fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487), 7ab12b4d6fb741a0aab3f5ee7bda9689 (127.22.230.61:43577)
I20260619 02:57:11.186158 30925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:11.186662 30200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bb2a01f104647df976f364cdd0f0dd2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689"
I20260619 02:57:11.186866 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.186264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42518)
0619 02:57:11.186633 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.186708 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
I20260619 02:57:11.186976 30200 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0bb2a01f104647df976f364cdd0f0dd2 in current term 2: Already voted for candidate 7ab12b4d6fb741a0aab3f5ee7bda9689 in this term.
I20260619 02:57:11.186985 30925 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 2 election: Requested vote from peers fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487), 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60:38553)
I20260619 02:57:11.187538 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.187143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42534)
0619 02:57:11.187363 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.187434 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 02:57:11.187495 30393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0bb2a01f104647df976f364cdd0f0dd2"
I20260619 02:57:11.187762 30393 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ab12b4d6fb741a0aab3f5ee7bda9689 in current term 2: Already voted for candidate 0bb2a01f104647df976f364cdd0f0dd2 in this term.
I20260619 02:57:11.188215 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.187428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42536)
0619 02:57:11.187999 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.188097 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
I20260619 02:57:11.188365 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.188056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42548)
0619 02:57:11.188188 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.188267 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:57:11.188377 30366 leader_election.cc:334] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
W20260619 02:57:11.188594 30174 leader_election.cc:334] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:11.188704 30174 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ab12b4d6fb741a0aab3f5ee7bda9689; no voters: 0bb2a01f104647df976f364cdd0f0dd2, fe80c7302c424ac0bb2ce35787ef3f60
I20260619 02:57:11.188728 30367 leader_election.cc:302] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bb2a01f104647df976f364cdd0f0dd2; no voters: 7ab12b4d6fb741a0aab3f5ee7bda9689, fe80c7302c424ac0bb2ce35787ef3f60
I20260619 02:57:11.188922 30925 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 02:57:11.188933 30924 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 02:57:11.643409 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.642646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:44899)
0619 02:57:11.643134 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.643254 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":253,"threads_started":1}
I20260619 02:57:11.648619 30935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.647998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:46189)
0619 02:57:11.648394 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.648475 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":223,"threads_started":1}
I20260619 02:57:11.666390 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:11.665838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:57955)
0619 02:57:11.666184 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:11.666252 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":186,"threads_started":1}
I20260619 02:57:12.644738 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.644039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:35033)
0619 02:57:12.644515 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.644586 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":288,"threads_started":1}
I20260619 02:57:12.649870 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.649260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:44239)
0619 02:57:12.649661 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.649728 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":228,"threads_started":1}
I20260619 02:57:12.667833 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.667233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:39957)
0619 02:57:12.667582 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.667666 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":178,"threads_started":1}
I20260619 02:57:12.799167 30940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:12.799373 30940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:12.800024 30940 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487), 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60:38553)
I20260619 02:57:12.800588 30393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0bb2a01f104647df976f364cdd0f0dd2" is_pre_election: true
I20260619 02:57:12.800813 30393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ab12b4d6fb741a0aab3f5ee7bda9689 in term 2.
I20260619 02:57:12.800907 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.800155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42894)
0619 02:57:12.800572 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.800695 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":181,"threads_started":1}
I20260619 02:57:12.801295 30177 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb2a01f104647df976f364cdd0f0dd2, 7ab12b4d6fb741a0aab3f5ee7bda9689; no voters:
I20260619 02:57:12.801587 30940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 02:57:12.801702 30940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:12.802055 30940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:57:12.802237 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.801885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42904)
0619 02:57:12.801996 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.802110 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 02:57:12.802376 30174 leader_election.cc:334] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:12.804979 30940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:12.805573 30940 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 3 election: Requested vote from peers fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487), 0bb2a01f104647df976f364cdd0f0dd2 (127.22.230.60:38553)
I20260619 02:57:12.806010 30393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0bb2a01f104647df976f364cdd0f0dd2"
I20260619 02:57:12.806124 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.805744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42918)
0619 02:57:12.805902 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.805987 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:57:12.806216 30393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:57:12.807035 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.806686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42922)
0619 02:57:12.806823 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.806896 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:57:12.807195 30174 leader_election.cc:334] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
I20260619 02:57:12.808481 30393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ab12b4d6fb741a0aab3f5ee7bda9689 in term 3.
I20260619 02:57:12.808876 30177 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bb2a01f104647df976f364cdd0f0dd2, 7ab12b4d6fb741a0aab3f5ee7bda9689; no voters: fe80c7302c424ac0bb2ce35787ef3f60
I20260619 02:57:12.809106 30940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:57:12.809613 30940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [term 3 LEADER]: Becoming Leader. State: Replica: 7ab12b4d6fb741a0aab3f5ee7bda9689, State: Running, Role: LEADER
I20260619 02:57:12.810053 30940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } }
I20260619 02:57:12.812633 30943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab12b4d6fb741a0aab3f5ee7bda9689. Latest consensus state: current_term: 3 leader_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:12.812871 30943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:12.813450 30945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:12.816969 30945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:12.817847 30945 catalog_manager.cc:1259] Loaded cluster ID: f220dbd2e48f42848627ebe0b42712d8
I20260619 02:57:12.817945 30945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:12.819020 30945 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:12.819758 30945 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Loaded TSK: 0
I20260619 02:57:12.820519 30945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:57:12.884127 30190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:57:12.890995 30190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:12.895025 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.894673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42954)
0619 02:57:12.894829 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.894897 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 02:57:12.895012 30393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [term 3 FOLLOWER]: Refusing update from remote peer 7ab12b4d6fb741a0aab3f5ee7bda9689: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 02:57:12.895677 30943 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 02:57:12.896076 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.895703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42956)
0619 02:57:12.895845 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.895970 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:57:12.896271 30174 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 -> Peer fe80c7302c424ac0bb2ce35787ef3f60 (127.22.230.62:38487): Couldn't send request to peer fe80c7302c424ac0bb2ce35787ef3f60. Status: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:57:12.898000 30947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab12b4d6fb741a0aab3f5ee7bda9689. Latest consensus state: current_term: 3 leader_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:12.898226 30947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:57:12.899609 30943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:12.899859 30943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:12.900556 30947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fe80c7302c424ac0bb2ce35787ef3f60" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38487 } } peers { permanent_uuid: "7ab12b4d6fb741a0aab3f5ee7bda9689" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43577 } attrs { promote: false } } peers { permanent_uuid: "0bb2a01f104647df976f364cdd0f0dd2" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 38553 } attrs { promote: false } } }
I20260619 02:57:12.900800 30947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb2a01f104647df976f364cdd0f0dd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:57:12.917197 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.907426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40529 (local address 127.0.0.1:43172)
0619 02:57:12.907582 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.907601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:12.907734 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:12.908801 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:12.908809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:12.908843 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:12.909185 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:12.909201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.911239 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.911247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:12.913965 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:12.913984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.916118 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.916127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:12.916386 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:12.916444 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 02:57:12.916522 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:57:12.916703 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:57:12.918955 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.907445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.22.230.3:40529)
0619 02:57:12.908422 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:12.908428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:12.908449 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:57:12.908522 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:12.908526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:12.908618 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:12.909355 (+ 737us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:12.909583 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.911043 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.914213 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.915877 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.918097 (+ 2220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:12.918264 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:12.918326 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 02:57:12.918502 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":1023,"threads_started":1}
I20260619 02:57:12.919831 30786 tablet_service.cc:1467] Processing CreateTablet for tablet 680da4f2a5004a149716404b10c5530b (DEFAULT_TABLE table=test-table [id=0af839ca56bf45938e0c95b2c1c82308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:12.920302 30786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680da4f2a5004a149716404b10c5530b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:12.927269 30958 tablet_bootstrap.cc:492] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Bootstrap starting.
I20260619 02:57:12.928040 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.908197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:34311 (local address 127.0.0.1:52640)
0619 02:57:12.917492 (+ 9295us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.917515 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:12.917627 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:12.922483 (+ 4856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:12.922489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:12.922518 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:12.922730 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:12.922745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.923939 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.923943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:12.925868 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:12.925884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.927480 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.927486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:12.927641 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:12.927674 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:57:12.927739 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:57:12.927820 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9180,"mutex_wait_us":1,"thread_start_us":11283,"threads_started":1}
I20260619 02:57:12.928900 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.908341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.22.230.2:34311)
0619 02:57:12.922220 (+ 13879us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:12.922225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:12.922240 (+ 15us) server_negotiation.cc:407] Connection header received
0619 02:57:12.922290 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:12.922294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:12.922348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:12.922458 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:12.922932 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.923797 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.926042 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.927314 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.928280 (+ 966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:12.928381 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:12.928428 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 02:57:12.928558 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13780,"thread_start_us":13696,"threads_started":1}
I20260619 02:57:12.929008 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.907799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:34993 (local address 127.0.0.1:48034)
0619 02:57:12.914579 (+ 6780us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.914603 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:12.914734 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:12.923493 (+ 8759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:12.923497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:12.923521 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:12.923805 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:12.923815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.924748 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.924754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:12.926659 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:12.926670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.928200 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.928208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:12.928371 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:12.928404 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:57:12.928476 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 02:57:12.928587 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6618,"mutex_wait_us":4,"thread_start_us":6497,"threads_started":1}
I20260619 02:57:12.929392 30958 tablet_bootstrap.cc:654] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:12.929666 30645 tablet_service.cc:1467] Processing CreateTablet for tablet 680da4f2a5004a149716404b10c5530b (DEFAULT_TABLE table=test-table [id=0af839ca56bf45938e0c95b2c1c82308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:12.929901 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.908202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.22.230.1:34993)
0619 02:57:12.923208 (+ 15006us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:12.923214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:12.923232 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:57:12.923292 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:12.923296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:12.923358 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:12.923452 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:12.923995 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.924621 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.926807 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:12.927980 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:12.928280 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:12.929425 (+ 1145us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:12.929474 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 02:57:12.929598 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14887,"thread_start_us":14777,"threads_started":1}
I20260619 02:57:12.930104 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680da4f2a5004a149716404b10c5530b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:12.930908 30497 tablet_service.cc:1467] Processing CreateTablet for tablet 680da4f2a5004a149716404b10c5530b (DEFAULT_TABLE table=test-table [id=0af839ca56bf45938e0c95b2c1c82308]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:12.931309 30497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680da4f2a5004a149716404b10c5530b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:12.931478 30958 tablet_bootstrap.cc:492] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: No bootstrap required, opened a new log
I20260619 02:57:12.931612 30958 ts_tablet_manager.cc:1397] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:57:12.932377 30958 raft_consensus.cc:348] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:12.932844 30958 raft_consensus.cc:374] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:12.933331 30958 raft_consensus.cc:729] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bad7dcc83b4d3d9f455747f7b85970, State: Initialized, Role: FOLLOWER
I20260619 02:57:12.933689 30958 consensus_queue.cc:260] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:12.935119 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.934738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:35923)
0619 02:57:12.934881 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.934971 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 02:57:12.935127 30958 ts_tablet_manager.cc:1428] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:12.936791 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.936368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:60269)
0619 02:57:12.936555 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.936634 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:57:12.937708 30853 heartbeater.cc:503] Master 127.22.230.61:43577 was elected leader, sending a full tablet report...
I20260619 02:57:12.939353 30962 tablet_bootstrap.cc:492] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Bootstrap starting.
I20260619 02:57:12.939576 30961 tablet_bootstrap.cc:492] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Bootstrap starting.
I20260619 02:57:12.941183 30962 tablet_bootstrap.cc:654] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:12.941458 30961 tablet_bootstrap.cc:654] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:12.943343 30962 tablet_bootstrap.cc:492] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: No bootstrap required, opened a new log
I20260619 02:57:12.943475 30962 ts_tablet_manager.cc:1397] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:57:12.943707 30961 tablet_bootstrap.cc:492] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: No bootstrap required, opened a new log
I20260619 02:57:12.943857 30961 ts_tablet_manager.cc:1397] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 02:57:12.944212 30962 raft_consensus.cc:348] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:12.944394 30962 raft_consensus.cc:374] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:12.944486 30962 raft_consensus.cc:729] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d12dd94a5c44168357171a07737412, State: Initialized, Role: FOLLOWER
I20260619 02:57:12.944598 30961 raft_consensus.cc:348] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:12.944792 30961 raft_consensus.cc:374] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:12.944722 30962 consensus_queue.cc:260] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:12.944854 30961 raft_consensus.cc:729] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4baead2a72f46ac962b668e105f2c7e, State: Initialized, Role: FOLLOWER
I20260619 02:57:12.945042 30961 consensus_queue.cc:260] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:12.945502 30962 ts_tablet_manager.cc:1428] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:57:12.946360 30961 ts_tablet_manager.cc:1428] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:57:12.946532 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.946093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:53401)
0619 02:57:12.946298 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.946390 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 02:57:12.947031 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.946595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:42257)
0619 02:57:12.946800 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.946880 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 02:57:12.947911 30564 heartbeater.cc:503] Master 127.22.230.61:43577 was elected leader, sending a full tablet report...
I20260619 02:57:12.947957 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:12.947598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:33277)
0619 02:57:12.947731 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:12.947808 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 02:57:12.950014 30712 heartbeater.cc:503] Master 127.22.230.61:43577 was elected leader, sending a full tablet report...
W20260619 02:57:12.988981 30565 tablet.cc:2369] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:57:13.022357 30960 raft_consensus.cc:482] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:13.022557 30960 raft_consensus.cc:504] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:13.023999 30960 leader_election.cc:288] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993), b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:13.031562 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.024463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:54925 (local address 127.22.230.2:34311)
0619 02:57:13.024678 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:13.024687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:13.025169 (+ 482us) server_negotiation.cc:407] Connection header received
0619 02:57:13.025312 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:13.025315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:13.025368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:13.025428 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:13.025858 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.026826 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.029269 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.030479 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.030667 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.030737 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:13.031179 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 02:57:13.031302 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:57:13.031560 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.024307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:51457 (local address 127.22.230.1:34993)
0619 02:57:13.024526 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:13.024534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:13.024558 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:57:13.024821 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:13.024827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:13.024903 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:13.025007 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:13.025525 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.026548 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.029287 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.030481 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.030678 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.030737 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:13.031179 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 02:57:13.031302 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:57:13.031649 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.024226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:34993 (local address 127.22.230.3:51457)
0619 02:57:13.024476 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.024504 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:13.024632 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:13.025044 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:13.025054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:13.025082 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:13.025361 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:13.025370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.026713 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.026717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:13.029132 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:13.029143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.030706 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.030713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:13.031009 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.031054 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 02:57:13.031142 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 02:57:13.031308 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":253}
I20260619 02:57:13.031713 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.024366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:34311 (local address 127.22.230.3:54925)
0619 02:57:13.025063 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.025077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:13.025191 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:13.025483 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:13.025490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:13.025515 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:13.025732 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:13.025738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.026959 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.026963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:13.029129 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:13.029143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.030706 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.030713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:13.030998 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.031042 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:57:13.031142 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:57:13.031308 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":132,"thread_start_us":392,"threads_started":1}
I20260619 02:57:13.032366 30517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680da4f2a5004a149716404b10c5530b" candidate_uuid: "48bad7dcc83b4d3d9f455747f7b85970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d12dd94a5c44168357171a07737412" is_pre_election: true
I20260619 02:57:13.032455 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680da4f2a5004a149716404b10c5530b" candidate_uuid: "48bad7dcc83b4d3d9f455747f7b85970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4baead2a72f46ac962b668e105f2c7e" is_pre_election: true
I20260619 02:57:13.032698 30665 raft_consensus.cc:2454] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48bad7dcc83b4d3d9f455747f7b85970 in term 0.
I20260619 02:57:13.032725 30517 raft_consensus.cc:2454] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48bad7dcc83b4d3d9f455747f7b85970 in term 0.
I20260619 02:57:13.033237 30741 leader_election.cc:302] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48bad7dcc83b4d3d9f455747f7b85970, b4baead2a72f46ac962b668e105f2c7e, c2d12dd94a5c44168357171a07737412; no voters:
I20260619 02:57:13.033603 30960 raft_consensus.cc:2781] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:13.033735 30960 raft_consensus.cc:482] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:13.033829 30960 raft_consensus.cc:3037] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:13.035789 30960 raft_consensus.cc:504] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:13.036302 30960 leader_election.cc:288] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [CANDIDATE]: Term 1 election: Requested vote from peers c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993), b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311)
I20260619 02:57:13.036687 30517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680da4f2a5004a149716404b10c5530b" candidate_uuid: "48bad7dcc83b4d3d9f455747f7b85970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2d12dd94a5c44168357171a07737412"
I20260619 02:57:13.036723 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680da4f2a5004a149716404b10c5530b" candidate_uuid: "48bad7dcc83b4d3d9f455747f7b85970" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4baead2a72f46ac962b668e105f2c7e"
I20260619 02:57:13.036863 30517 raft_consensus.cc:3037] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:13.036916 30665 raft_consensus.cc:3037] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:13.038910 30517 raft_consensus.cc:2454] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48bad7dcc83b4d3d9f455747f7b85970 in term 1.
I20260619 02:57:13.038920 30665 raft_consensus.cc:2454] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48bad7dcc83b4d3d9f455747f7b85970 in term 1.
I20260619 02:57:13.039307 30739 leader_election.cc:302] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48bad7dcc83b4d3d9f455747f7b85970, c2d12dd94a5c44168357171a07737412; no voters:
I20260619 02:57:13.039642 30960 raft_consensus.cc:2781] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:13.040022 30960 raft_consensus.cc:686] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 LEADER]: Becoming Leader. State: Replica: 48bad7dcc83b4d3d9f455747f7b85970, State: Running, Role: LEADER
I20260619 02:57:13.040441 30960 consensus_queue.cc:237] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } }
I20260619 02:57:13.040982 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.040616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:37287)
0619 02:57:13.040789 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.040856 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 02:57:13.044149 30190 catalog_manager.cc:5495] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3). New cstate: current_term: 1 leader_uuid: "48bad7dcc83b4d3d9f455747f7b85970" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bad7dcc83b4d3d9f455747f7b85970" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 } health_report { overall_health: UNKNOWN } } }
W20260619 02:57:13.059805 30713 tablet.cc:2369] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:57:13.081847 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.077270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.22.230.3:40529)
0619 02:57:13.077436 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:13.077441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:13.077457 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:13.077579 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:13.077583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:13.077660 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:13.077766 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:13.078044 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.078882 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.080108 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.080464 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.080547 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.080878 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:13.081396 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:13.081494 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:13.081601 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 02:57:13.081658 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:57:13.085539 30665 raft_consensus.cc:1261] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Refusing update from remote peer 48bad7dcc83b4d3d9f455747f7b85970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:13.085812 30517 raft_consensus.cc:1261] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Refusing update from remote peer 48bad7dcc83b4d3d9f455747f7b85970: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:13.086475 30968 consensus_queue.cc:1035] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4baead2a72f46ac962b668e105f2c7e" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:13.086850 30960 consensus_queue.cc:1035] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d12dd94a5c44168357171a07737412" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:13.088969 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.088536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:49987)
0619 02:57:13.088692 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.088787 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 02:57:13.089823 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.089222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:58815)
0619 02:57:13.089455 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.089552 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260619 02:57:13.095578 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.095033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.1:57607)
0619 02:57:13.095287 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.095395 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 02:57:13.101536 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.101129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:48629)
0619 02:57:13.101276 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.101361 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 02:57:13.105676 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.105065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.2:51441)
0619 02:57:13.105372 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.105486 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 02:57:13.106366 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.105600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.22.230.3:44953)
0619 02:57:13.106121 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.106197 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
I20260619 02:57:13.115758 30189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:13.116051 30189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:13.121456 30189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ab12b4d6fb741a0aab3f5ee7bda9689: Sending DeleteTablet for 3 replicas of tablet 680da4f2a5004a149716404b10c5530b
I20260619 02:57:13.122416 30786 tablet_service.cc:1514] Processing DeleteTablet for tablet 680da4f2a5004a149716404b10c5530b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:13 UTC) from {username='oryx', principal='oryx/127.22.230.61@KRBTEST.COM'} at 127.0.0.1:43172
I20260619 02:57:13.122394 30497 tablet_service.cc:1514] Processing DeleteTablet for tablet 680da4f2a5004a149716404b10c5530b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:13 UTC) from {username='oryx', principal='oryx/127.22.230.61@KRBTEST.COM'} at 127.0.0.1:48034
I20260619 02:57:13.123234 30980 tablet_replica.cc:330] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: stopping tablet replica
I20260619 02:57:13.123452 30980 raft_consensus.cc:2229] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:13.124051 30980 raft_consensus.cc:2258] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:13.125195 30980 ts_tablet_manager.cc:1905] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:13.130956 30981 tablet_replica.cc:330] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: stopping tablet replica
I20260619 02:57:13.130949 30645 tablet_service.cc:1514] Processing DeleteTablet for tablet 680da4f2a5004a149716404b10c5530b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:13 UTC) from {username='oryx', principal='oryx/127.22.230.61@KRBTEST.COM'} at 127.0.0.1:52640
I20260619 02:57:13.131178 30981 raft_consensus.cc:2229] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:13.131444 30981 raft_consensus.cc:2258] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:13.131964 30980 ts_tablet_manager.cc:1918] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:13.132150 30980 log.cc:1196] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/680da4f2a5004a149716404b10c5530b
I20260619 02:57:13.132704 30980 ts_tablet_manager.cc:1939] T 680da4f2a5004a149716404b10c5530b P 48bad7dcc83b4d3d9f455747f7b85970: Deleting consensus metadata
I20260619 02:57:13.133597 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 30370
I20260619 02:57:13.133843 30175 catalog_manager.cc:4841] TS 48bad7dcc83b4d3d9f455747f7b85970 (127.22.230.3:40529): tablet 680da4f2a5004a149716404b10c5530b (table test-table [id=0af839ca56bf45938e0c95b2c1c82308]) successfully deleted
I20260619 02:57:13.137511 30981 ts_tablet_manager.cc:1905] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:13.150468 30981 ts_tablet_manager.cc:1918] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:13.153820 30981 log.cc:1196] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/680da4f2a5004a149716404b10c5530b
I20260619 02:57:13.155298 30981 ts_tablet_manager.cc:1939] T 680da4f2a5004a149716404b10c5530b P c2d12dd94a5c44168357171a07737412: Deleting consensus metadata
I20260619 02:57:13.157699 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.134789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.22.230.1:34993)
0619 02:57:13.134946 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:13.134952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:13.140442 (+ 5490us) server_negotiation.cc:407] Connection header received
0619 02:57:13.140634 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:13.140642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:13.140738 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:13.140830 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:13.142741 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.143885 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.145849 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.146536 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.146658 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.150257 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:13.150275 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:13.150295 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:13.150351 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:13.154277 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:13.155136 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:13.155145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:13.155149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:13.155393 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:13.155863 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:13.155869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:13.155872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:13.156318 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:13.156583 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:13.157146 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 02:57:13.157350 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:57:13.159214 30557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:13.160274 30982 tablet_replica.cc:330] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: stopping tablet replica
I20260619 02:57:13.160526 30982 raft_consensus.cc:2229] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:13.160887 30982 raft_consensus.cc:2258] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:13.161996 30982 ts_tablet_manager.cc:1905] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:13.169584 30982 ts_tablet_manager.cc:1918] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:13.169754 30982 log.cc:1196] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/680da4f2a5004a149716404b10c5530b
I20260619 02:57:13.170246 30982 ts_tablet_manager.cc:1939] T 680da4f2a5004a149716404b10c5530b P b4baead2a72f46ac962b668e105f2c7e: Deleting consensus metadata
I20260619 02:57:13.171398 30176 catalog_manager.cc:4841] TS b4baead2a72f46ac962b668e105f2c7e (127.22.230.2:34311): tablet 680da4f2a5004a149716404b10c5530b (table test-table [id=0af839ca56bf45938e0c95b2c1c82308]) successfully deleted
I20260619 02:57:13.285322 30174 catalog_manager.cc:4841] TS c2d12dd94a5c44168357171a07737412 (127.22.230.1:34993): tablet 680da4f2a5004a149716404b10c5530b (table test-table [id=0af839ca56bf45938e0c95b2c1c82308]) successfully deleted
I20260619 02:57:13.292549 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 30370
I20260619 02:57:13.321755 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 30574
I20260619 02:57:13.334800 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.322526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.22.230.2:34311)
0619 02:57:13.322698 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:13.322705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:13.322800 (+ 95us) server_negotiation.cc:407] Connection header received
0619 02:57:13.322986 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:13.322997 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:13.323057 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:13.323170 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:13.324331 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.325036 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.326224 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.326725 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.326858 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.329310 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:13.329323 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:13.329337 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:13.329366 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:13.332112 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:13.332810 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:13.332817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:13.332819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:13.332939 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:13.333288 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:13.333294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:13.333297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:13.333608 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:13.333927 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:13.334171 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 02:57:13.334383 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 02:57:13.335544 30705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:13.429610 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 30574
I20260619 02:57:13.452979 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 30723
I20260619 02:57:13.466917 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.453809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.22.230.3:40529)
0619 02:57:13.453982 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:13.453988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:13.454009 (+ 21us) server_negotiation.cc:407] Connection header received
0619 02:57:13.454235 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:13.454240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:13.454334 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:13.454447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:13.455983 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.456754 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.457944 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:13.458394 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:13.458538 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:13.461058 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:13.461073 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:13.461091 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:13.461126 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:13.464202 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:13.464881 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:13.464896 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:13.464899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:13.465054 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:13.465428 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:13.465434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:13.465437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:13.465757 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:13.466112 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:13.466334 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 02:57:13.466532 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 02:57:13.467612 30846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:13.483397 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.470777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42962)
0619 02:57:13.483013 (+ 12236us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.483153 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12085,"thread_start_us":11950,"threads_started":1}
I20260619 02:57:13.484817 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:13.484434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38487 (local address 127.0.0.1:42974)
0619 02:57:13.484598 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:13.484685 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 02:57:13.588562 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 30723
I20260619 02:57:13.609695 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 30053
I20260619 02:57:13.610822 30220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:13.728883 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 30053
I20260619 02:57:13.750108 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 30153
I20260619 02:57:13.751174 30413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:13.861135 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 30153
2026-06-19T02:57:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10784 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:57:13 dist-test-slave-2gms krb5kdc[30993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:57:13 dist-test-slave-2gms krb5kdc[30993](info): set up 2 sockets
Jun 19 02:57:13 dist-test-slave-2gms krb5kdc[30993](info): commencing operation
krb5kdc: starting...
W20260619 02:57:14.951479 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:15 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab.
Entry for principal oryx/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab.
I20260619 02:57:15.153680 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:46437
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab
--principal=oryx/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:15.457523 31006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:15.457765 31006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:15.457849 31006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:15.464732 31006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:15.464823 31006 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 02:57:15.464871 31006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:15.464928 31006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:15.464980 31006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:15.465039 31006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:15.474177 31006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:46437
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.62.keytab
--principal=oryx/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:15.475273 31006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:15.476473 31006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:15.487741 31014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:15.487803 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:15.487830 31012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:15.488389 31006 server_base.cc:1034] running on GCE node
I20260619 02:57:15.489138 31006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:15.490471 31006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:15.491683 31006 hybrid_clock.cc:648] HybridClock initialized: now 1781837835491619 us; error 37 us; skew 500 ppm
I20260619 02:57:15.492014 31006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:15 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837835, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:15.494994 31006 init.cc:377] Logged in from keytab as oryx/127.22.230.62@KRBTEST.COM (short username oryx)
I20260619 02:57:15.496273 31006 webserver.cc:460] Webserver started at http://127.22.230.62:35509/ using document root <none> and password file <none>
I20260619 02:57:15.496901 31006 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:15.496981 31006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:15.497231 31006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:15.500061 31006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "1d24fc5772484d1686a0843c6906fb58"
format_stamp: "Formatted at 2026-06-19 02:57:15 on dist-test-slave-2gms"
server_key: "99197e0972382f2a3fbd74f97dd8aec5"
server_key_iv: "6ed51e0e7a6a76e7d81c44d2273e32b4"
server_key_version: "encryptionkey@0"
I20260619 02:57:15.500823 31006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "1d24fc5772484d1686a0843c6906fb58"
format_stamp: "Formatted at 2026-06-19 02:57:15 on dist-test-slave-2gms"
server_key: "99197e0972382f2a3fbd74f97dd8aec5"
server_key_iv: "6ed51e0e7a6a76e7d81c44d2273e32b4"
server_key_version: "encryptionkey@0"
I20260619 02:57:15.505643 31006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 02:57:15.508854 31021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:15.510080 31006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:57:15.510247 31006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "1d24fc5772484d1686a0843c6906fb58"
format_stamp: "Formatted at 2026-06-19 02:57:15 on dist-test-slave-2gms"
server_key: "99197e0972382f2a3fbd74f97dd8aec5"
server_key_iv: "6ed51e0e7a6a76e7d81c44d2273e32b4"
server_key_version: "encryptionkey@0"
I20260619 02:57:15.510452 31006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:15.536702 31006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:15.540413 31006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:15.540676 31006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:15.561277 31006 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:46437
I20260619 02:57:15.561359 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:46437 every 8 connection(s)
I20260619 02:57:15.562538 31006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:15.566116 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:15.571733 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31006
I20260619 02:57:15.571929 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:15.572549 23448 external_mini_cluster.cc:1428] Setting key b33354235812050015975ed357f284ef
I20260619 02:57:15.576035 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Bootstrap starting.
I20260619 02:57:15.579605 31083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:15.580593 31083 log.cc:824] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:15.583346 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: No bootstrap required, opened a new log
Jun 19 02:57:15 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:15.593523 31083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d24fc5772484d1686a0843c6906fb58" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46437 } }
I20260619 02:57:15.594022 31083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:15.594129 31083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d24fc5772484d1686a0843c6906fb58, State: Initialized, Role: FOLLOWER
I20260619 02:57:15.594795 31083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d24fc5772484d1686a0843c6906fb58" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46437 } }
I20260619 02:57:15.595064 31083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:15.595212 31083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:15.595383 31083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:15.595502 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:15.574374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.22.230.62:46437)
0619 02:57:15.575134 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:15.575144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:15.575183 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:57:15.576800 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:15.576820 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:15.577169 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:15.577504 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:15.579251 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:15.580628 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:15.582197 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:15.582871 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:15.582992 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:15.587460 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:15.587469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:15.587487 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:15.587535 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:15.591371 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:15.591976 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:15.591985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:15.591993 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:15.592195 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:15.592511 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:15.592516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:15.592520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:15.593154 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:15.593411 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:15.593757 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 02:57:15.594187 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":384,"threads_started":1}
I20260619 02:57:15.597653 31083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d24fc5772484d1686a0843c6906fb58" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46437 } }
I20260619 02:57:15.598183 31083 leader_election.cc:302] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d24fc5772484d1686a0843c6906fb58; no voters:
I20260619 02:57:15.598766 31083 leader_election.cc:288] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:15.598857 31088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:15.599613 31088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [term 1 LEADER]: Becoming Leader. State: Replica: 1d24fc5772484d1686a0843c6906fb58, State: Running, Role: LEADER
I20260619 02:57:15.600173 31088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d24fc5772484d1686a0843c6906fb58" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46437 } }
I20260619 02:57:15.600734 31083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:15.602996 31089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d24fc5772484d1686a0843c6906fb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d24fc5772484d1686a0843c6906fb58" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46437 } } }
I20260619 02:57:15.603309 31089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:15.604782 31096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:15.604852 31090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d24fc5772484d1686a0843c6906fb58. Latest consensus state: current_term: 1 leader_uuid: "1d24fc5772484d1686a0843c6906fb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d24fc5772484d1686a0843c6906fb58" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 46437 } } }
I20260619 02:57:15.605119 31090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:15.609562 31096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:15.618259 31096 catalog_manager.cc:1347] Generated new cluster ID: c463c42e906e4fa0a3d2448f6b83aafb
I20260619 02:57:15.618412 31096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:15.644105 31096 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:15.645412 31096 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:15.655733 31096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Generated new TSK 0
I20260619 02:57:15.656502 31096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab.
Entry for principal oryx/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab.
I20260619 02:57:15.714238 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46437
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:16.049124 31109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:16.049376 31109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:16.049465 31109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:16.056169 31109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:16.056260 31109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:16.056339 31109 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:57:16.056414 31109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:16.066831 31109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:46437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:16.068004 31109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:16.069195 31109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:16.081799 31114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:16.081847 31115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:16.082037 31117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:16.082795 31109 server_base.cc:1034] running on GCE node
I20260619 02:57:16.083433 31109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:16.084280 31109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:16.085500 31109 hybrid_clock.cc:648] HybridClock initialized: now 1781837836085444 us; error 31 us; skew 500 ppm
I20260619 02:57:16.085788 31109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:16 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837836, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:16.088871 31109 init.cc:377] Logged in from keytab as oryx/127.22.230.1@KRBTEST.COM (short username oryx)
I20260619 02:57:16.090116 31109 webserver.cc:460] Webserver started at http://127.22.230.1:34731/ using document root <none> and password file <none>
I20260619 02:57:16.090870 31109 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:16.090966 31109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:16.091250 31109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:16.094048 31109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "db706169f8d4423abb7b8b5b52703c95"
format_stamp: "Formatted at 2026-06-19 02:57:16 on dist-test-slave-2gms"
server_key: "d0ea5b8ba95488c572f1216e63d8387d"
server_key_iv: "9a8427adfc77317c8c67a5b70af6f277"
server_key_version: "encryptionkey@0"
I20260619 02:57:16.094867 31109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "db706169f8d4423abb7b8b5b52703c95"
format_stamp: "Formatted at 2026-06-19 02:57:16 on dist-test-slave-2gms"
server_key: "d0ea5b8ba95488c572f1216e63d8387d"
server_key_iv: "9a8427adfc77317c8c67a5b70af6f277"
server_key_version: "encryptionkey@0"
I20260619 02:57:16.099917 31109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 02:57:16.103458 31124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:16.104653 31109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:57:16.104832 31109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "db706169f8d4423abb7b8b5b52703c95"
format_stamp: "Formatted at 2026-06-19 02:57:16 on dist-test-slave-2gms"
server_key: "d0ea5b8ba95488c572f1216e63d8387d"
server_key_iv: "9a8427adfc77317c8c67a5b70af6f277"
server_key_version: "encryptionkey@0"
I20260619 02:57:16.105031 31109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:16.133265 31109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:16.137025 31109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:16.137306 31109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:16.139827 31109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:16.139931 31109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:16.140112 31109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:16.140234 31109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 02:57:16 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837836, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:16.163566 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.140289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:52891 (local address 127.22.230.62:46437)
0619 02:57:16.140728 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:16.140734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:16.141969 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 02:57:16.144197 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:16.144202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:16.144285 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:16.144421 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:16.146623 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.147986 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.150530 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.151199 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.151332 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.156649 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:16.156655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:16.156659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:16.156689 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:16.159553 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.160594 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.160600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.160606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.160726 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.161341 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.161345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.161347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.161660 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:16.161802 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:16.162884 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0619 02:57:16.163133 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
I20260619 02:57:16.165472 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.140954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.22.230.1:52891)
0619 02:57:16.141783 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:16.141833 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:16.143926 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:16.144660 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:16.144673 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:16.145496 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:16.146340 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.146358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.148185 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.148193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.150385 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.150396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.151443 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.151450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.151737 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.152619 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:16.152667 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:16.156406 (+ 3739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:16.159705 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.159716 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.159737 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.160454 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.160856 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.160864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.160868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.161179 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.161842 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:16.161854 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:16.162390 (+ 536us) client_negotiation.cc:769] Sending connection context
0619 02:57:16.162793 (+ 403us) client_negotiation.cc:240] Negotiation successful
0619 02:57:16.163446 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":354,"threads_started":1}
I20260619 02:57:16.184751 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.171908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:59693 (local address 127.22.230.62:46437)
0619 02:57:16.172119 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:16.172127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:16.172683 (+ 556us) server_negotiation.cc:407] Connection header received
0619 02:57:16.172890 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:16.172897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:16.172992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:16.173100 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:16.174451 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.175374 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.176636 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.177158 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.177312 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.179418 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:16.179424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:16.179429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:16.179465 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:16.181744 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.182351 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.182356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.182360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.182478 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.182856 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.182860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.182864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.183238 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:16.183369 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:16.184188 (+ 819us) server_negotiation.cc:299] Negotiation successful
0619 02:57:16.184415 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:57:16.184842 31207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.171984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.22.230.1:59693)
0619 02:57:16.172572 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:16.172596 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:16.172759 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:16.173176 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:16.173183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:16.173814 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:16.174314 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.174324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.175504 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.175509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.176494 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.176504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.177644 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.177662 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.177780 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.178473 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:16.178488 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:16.179245 (+ 757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:16.181863 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.181870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.181874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.182235 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.182602 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.182607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.182611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.182741 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.183335 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:16.183340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:16.183567 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 02:57:16.183745 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 02:57:16.184129 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":317,"threads_started":1}
I20260619 02:57:16.197762 31109 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:41743
I20260619 02:57:16.197782 31244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:41743 every 8 connection(s)
I20260619 02:57:16.198954 31109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:16.200881 31245 heartbeater.cc:347] Connected to a master server at 127.22.230.62:46437
I20260619 02:57:16.201058 31245 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:16.201699 31245 heartbeater.cc:511] Master 127.22.230.62:46437 requested a full tablet report, sending...
I20260619 02:57:16.203436 31038 ts_manager.cc:194] Registered new tserver with Master: db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743)
I20260619 02:57:16.204809 31038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:52891
I20260619 02:57:16.205286 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31109
I20260619 02:57:16.205503 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:16.206218 23448 external_mini_cluster.cc:1428] Setting key fac071a1837ea2ef58db0b4449f21257
WARNING: no policy specified for oryx/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab.
Entry for principal oryx/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab.
I20260619 02:57:16.251520 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46437
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:16.565368 31249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:16.565698 31249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:16.565868 31249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:16.572974 31249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:16.573120 31249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:16.573258 31249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:57:16.573392 31249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:16.582553 31249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:46437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:16.583693 31249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:16.584913 31249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:16.598274 31255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:16.598336 31254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:16.598800 31257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:16.599097 31249 server_base.cc:1034] running on GCE node
I20260619 02:57:16.599740 31249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:16.600672 31249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:16.601877 31249 hybrid_clock.cc:648] HybridClock initialized: now 1781837836601806 us; error 39 us; skew 500 ppm
I20260619 02:57:16.602185 31249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:16 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837836, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:16.605142 31249 init.cc:377] Logged in from keytab as oryx/127.22.230.2@KRBTEST.COM (short username oryx)
I20260619 02:57:16.606426 31249 webserver.cc:460] Webserver started at http://127.22.230.2:45101/ using document root <none> and password file <none>
I20260619 02:57:16.607129 31249 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:16.607239 31249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:16.607555 31249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:16.610497 31249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "9f048aa6ee9e402884c63d21a526dfbd"
format_stamp: "Formatted at 2026-06-19 02:57:16 on dist-test-slave-2gms"
server_key: "85bb5592b811857d09b6ed91e184afc6"
server_key_iv: "9554c4b03ace7f3762f8605bf52336d3"
server_key_version: "encryptionkey@0"
I20260619 02:57:16.611356 31249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "9f048aa6ee9e402884c63d21a526dfbd"
format_stamp: "Formatted at 2026-06-19 02:57:16 on dist-test-slave-2gms"
server_key: "85bb5592b811857d09b6ed91e184afc6"
server_key_iv: "9554c4b03ace7f3762f8605bf52336d3"
server_key_version: "encryptionkey@0"
I20260619 02:57:16.616077 31249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:57:16.619297 31264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:16.620527 31249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:16.620692 31249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "9f048aa6ee9e402884c63d21a526dfbd"
format_stamp: "Formatted at 2026-06-19 02:57:16 on dist-test-slave-2gms"
server_key: "85bb5592b811857d09b6ed91e184afc6"
server_key_iv: "9554c4b03ace7f3762f8605bf52336d3"
server_key_version: "encryptionkey@0"
I20260619 02:57:16.620888 31249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:16.639525 31249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:16.643154 31249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:16.643404 31249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:16.645769 31249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:16.645879 31249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:16.646045 31249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:16.646132 31249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:57:16 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837836, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:16.668982 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.646506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:43245 (local address 127.22.230.62:46437)
0619 02:57:16.646707 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:16.646713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:16.648064 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 02:57:16.650024 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:16.650032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:16.650144 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:16.650263 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:16.652276 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.653006 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.655519 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.656136 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.656270 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.661379 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:16.661389 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:16.661396 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:16.661445 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:16.665542 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.666416 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.666421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.666427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.666618 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.667253 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.667260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.667265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.667523 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:16.667695 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:16.668421 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 02:57:16.668632 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 02:57:16.669939 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.647056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.22.230.2:43245)
0619 02:57:16.647891 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:16.647933 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:16.649769 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:16.650519 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:16.650532 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:16.651316 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:16.652104 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.652122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.653163 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.653172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.655366 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.655377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.656375 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.656386 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.656745 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.657495 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:16.657527 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:16.661055 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:16.665695 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.665701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.665711 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.666290 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.666766 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.666773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.666782 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.667093 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.667651 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:16.667660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:16.668002 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 02:57:16.668336 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 02:57:16.668813 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":360,"threads_started":1}
I20260619 02:57:16.690757 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.675357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:51685 (local address 127.22.230.62:46437)
0619 02:57:16.675597 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:16.675606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:16.676158 (+ 552us) server_negotiation.cc:407] Connection header received
0619 02:57:16.676294 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:16.676302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:16.676395 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:16.676520 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:16.677330 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.678799 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.680483 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.681044 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.681185 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.683973 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:16.683981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:16.683987 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:16.684025 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:16.687557 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.688426 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.688432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.688438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.688589 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:16.689009 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:16.689017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:16.689023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:16.689431 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:16.689611 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:16.690054 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 02:57:16.690306 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 02:57:16.690837 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:16.675442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.22.230.2:51685)
0619 02:57:16.676062 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:16.676077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:16.676185 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:16.676519 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:16.676522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:16.676908 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:16.677194 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.677202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.678959 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.678970 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.680341 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:16.680355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:16.681258 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:16.681270 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:16.681578 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:16.682358 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:16.682391 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:16.683739 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:16.687690 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.687698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.687702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.688240 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.688741 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:16.688745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:16.688747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:16.688865 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:16.689566 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:16.689570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:16.689791 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 02:57:16.689927 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 02:57:16.690149 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":349,"threads_started":1}
I20260619 02:57:16.700711 31249 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:42749
I20260619 02:57:16.700759 31383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:42749 every 8 connection(s)
I20260619 02:57:16.701926 31249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:16.702217 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31249
I20260619 02:57:16.702400 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:16.703063 23448 external_mini_cluster.cc:1428] Setting key af917fb8923baf57239cc7bbcbae85ec
I20260619 02:57:16.703924 31384 heartbeater.cc:347] Connected to a master server at 127.22.230.62:46437
I20260619 02:57:16.704144 31384 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:16.704866 31384 heartbeater.cc:511] Master 127.22.230.62:46437 requested a full tablet report, sending...
I20260619 02:57:16.706115 31038 ts_manager.cc:194] Registered new tserver with Master: 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749)
I20260619 02:57:16.707001 31038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:43245
WARNING: no policy specified for oryx/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab.
Entry for principal oryx/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab.
I20260619 02:57:16.747759 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:46437
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:17.055249 31388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:17.055547 31388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:17.055712 31388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:17.062618 31388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:17.062744 31388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:17.062887 31388 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 02:57:17.063022 31388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:17.072297 31388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.230.62:46437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:17.073493 31388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:17.074761 31388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:17.087356 31393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:17.087402 31396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:17.087345 31394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:17.088258 31388 server_base.cc:1034] running on GCE node
I20260619 02:57:17.088763 31388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:17.089514 31388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:17.090692 31388 hybrid_clock.cc:648] HybridClock initialized: now 1781837837090616 us; error 38 us; skew 500 ppm
I20260619 02:57:17.090976 31388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:17 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837837, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:17.093979 31388 init.cc:377] Logged in from keytab as oryx/127.22.230.3@KRBTEST.COM (short username oryx)
I20260619 02:57:17.096184 31388 webserver.cc:460] Webserver started at http://127.22.230.3:45281/ using document root <none> and password file <none>
I20260619 02:57:17.096853 31388 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:17.096974 31388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:17.097287 31388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:17.100051 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "1d5246855b934e5fbedbc99c6f5ac904"
format_stamp: "Formatted at 2026-06-19 02:57:17 on dist-test-slave-2gms"
server_key: "4174b578aa04c63003d02ce5a2c64121"
server_key_iv: "acefdcaf403654182af965576139581b"
server_key_version: "encryptionkey@0"
I20260619 02:57:17.100868 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "1d5246855b934e5fbedbc99c6f5ac904"
format_stamp: "Formatted at 2026-06-19 02:57:17 on dist-test-slave-2gms"
server_key: "4174b578aa04c63003d02ce5a2c64121"
server_key_iv: "acefdcaf403654182af965576139581b"
server_key_version: "encryptionkey@0"
I20260619 02:57:17.105652 31388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:57:17.108987 31403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:17.110308 31388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:17.110468 31388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "1d5246855b934e5fbedbc99c6f5ac904"
format_stamp: "Formatted at 2026-06-19 02:57:17 on dist-test-slave-2gms"
server_key: "4174b578aa04c63003d02ce5a2c64121"
server_key_iv: "acefdcaf403654182af965576139581b"
server_key_version: "encryptionkey@0"
I20260619 02:57:17.110759 31388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:17.134054 31388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:17.137599 31388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:17.137851 31388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:17.140410 31388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:17.140520 31388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:17.140703 31388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:17.140782 31388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 02:57:17 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837837, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:17.161880 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:17.140856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:38749 (local address 127.22.230.62:46437)
0619 02:57:17.141046 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:17.141055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:17.142321 (+ 1266us) server_negotiation.cc:407] Connection header received
0619 02:57:17.144402 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:17.144408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:17.144497 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:17.144628 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:17.145911 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.147248 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.150536 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.151143 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.151295 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:17.154986 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:17.154991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:17.154994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:17.155023 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:17.158298 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:17.159166 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:17.159175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:17.159181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:17.159387 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:17.159810 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:17.159818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:17.159824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:17.160250 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:17.160424 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:17.161079 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 02:57:17.161374 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:57:17.162254 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:17.141428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.22.230.3:38749)
0619 02:57:17.142159 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:17.142198 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:17.144068 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:17.144787 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:17.144794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:17.145214 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:17.145746 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:17.145762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.147459 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.147464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:17.150370 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:17.150393 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.151439 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.151447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:17.151632 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:17.152270 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:17.152290 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:17.154754 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:17.158429 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:17.158441 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:17.158455 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:17.159032 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:17.159514 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:17.159517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:17.159520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:17.159692 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:17.160377 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:17.160382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:17.160670 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 02:57:17.160870 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 02:57:17.161200 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":322,"threads_started":1}
I20260619 02:57:17.181399 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:17.168187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:46231 (local address 127.22.230.62:46437)
0619 02:57:17.168354 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:17.168358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:17.168988 (+ 630us) server_negotiation.cc:407] Connection header received
0619 02:57:17.169167 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:17.169172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:17.169238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:17.169332 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:17.170191 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.170954 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.172460 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.172892 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.173006 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:17.175067 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:17.175071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:17.175074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:17.175099 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:17.178341 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:17.179236 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:17.179242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:17.179248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:17.179368 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:17.179821 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:17.179828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:17.179834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:17.180235 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:17.180404 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:17.180837 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 02:57:17.181034 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:57:17.181631 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:17.168271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.22.230.3:46231)
0619 02:57:17.168897 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:17.168911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:17.169044 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:17.169350 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:17.169355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:17.169792 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:17.170060 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:17.170068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.171082 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.171087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:17.172310 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:17.172323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.173066 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.173073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:17.173250 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:17.173826 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:17.173842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:17.174871 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:17.178466 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:17.178472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:17.178476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:17.179100 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:17.179481 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:17.179488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:17.179492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:17.179698 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:17.180376 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:17.180381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:17.180577 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 02:57:17.180742 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 02:57:17.181059 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":375,"threads_started":1}
I20260619 02:57:17.196511 31388 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:43447
I20260619 02:57:17.196532 31522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:43447 every 8 connection(s)
I20260619 02:57:17.197696 31388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:17.199803 31523 heartbeater.cc:347] Connected to a master server at 127.22.230.62:46437
I20260619 02:57:17.200039 31523 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:17.200666 31523 heartbeater.cc:511] Master 127.22.230.62:46437 requested a full tablet report, sending...
I20260619 02:57:17.202028 31038 ts_manager.cc:194] Registered new tserver with Master: 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:17.202934 31038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:38749
I20260619 02:57:17.205410 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31388
I20260619 02:57:17.205541 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:17.205878 23448 external_mini_cluster.cc:1428] Setting key 6b5e9f52802eec1a29fa06cf88ec6b0b
I20260619 02:57:17.207499 31245 heartbeater.cc:503] Master 127.22.230.62:46437 was elected leader, sending a full tablet report...
I20260619 02:57:17.208716 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 02:57:17 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 02:57:17 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:17 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:17.709417 31384 heartbeater.cc:503] Master 127.22.230.62:46437 was elected leader, sending a full tablet report...
I20260619 02:57:18.204959 31523 heartbeater.cc:503] Master 127.22.230.62:46437 was elected leader, sending a full tablet report...
W20260619 02:57:18.294181 31531 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.043s user 0.000s sys 0.014s
W20260619 02:57:18.295423 31132 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:17.245846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.22.230.62:46437)
0619 02:57:17.246065 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:17.246073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:17.246338 (+ 265us) server_negotiation.cc:407] Connection header received
0619 02:57:17.246542 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:17.246547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:17.246677 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:17.246781 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:17.248057 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.248782 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.250003 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:17.250424 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:17.250576 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.295114 (+1044538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55802: BlockingRecv error: recv got EOF from 127.0.0.1:55802 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 02:57:18 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:18.318989 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.302913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.22.230.62:46437)
0619 02:57:18.303140 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.303145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.303330 (+ 185us) server_negotiation.cc:407] Connection header received
0619 02:57:18.303478 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.303482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.303568 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.303651 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:18.304949 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.305615 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.306857 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.307257 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.307351 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.313842 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:18.313847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:18.313850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:18.313896 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:18.316705 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:18.317314 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:18.317319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:18.317321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:18.317444 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:18.317752 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:18.317756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:18.317758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:18.318019 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:18.318225 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.318397 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.318559 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 02:57:18.323030 31037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:57:18.329838 31037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:18.354641 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.342103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:42749 (local address 127.0.0.1:52238)
0619 02:57:18.343104 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:18.343121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:18.343238 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:18.344337 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:18.344345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:18.344379 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:18.344939 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.344952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.347559 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.347568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.350842 (+ 3274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.350862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.353395 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.353405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.353674 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.353750 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 02:57:18.353876 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 02:57:18.354116 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":744,"threads_started":1}
I20260619 02:57:18.356225 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.342212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.22.230.2:42749)
0619 02:57:18.343705 (+ 1493us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.343712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.343737 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:18.343828 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.343834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.344085 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.345169 (+ 1084us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:18.345406 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.347355 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.351140 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.353116 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.355340 (+ 2224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.355625 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.355704 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.355915 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":1237,"threads_started":1}
I20260619 02:57:18.356720 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.341032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41743 (local address 127.0.0.1:48056)
0619 02:57:18.341726 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:18.341769 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:18.341944 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:18.343549 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:18.343556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:18.343582 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:18.344023 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.344037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.347645 (+ 3608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.347654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.351937 (+ 4283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.351956 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.355813 (+ 3857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.355822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.356074 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.356120 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:57:18.356233 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 02:57:18.356394 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":235,"threads_started":1}
I20260619 02:57:18.356894 31543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.341221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.22.230.1:41743)
0619 02:57:18.341786 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.341799 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.341831 (+ 32us) server_negotiation.cc:407] Connection header received
0619 02:57:18.342671 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.342684 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.343029 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.343250 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:18.344258 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.347443 (+ 3185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.352175 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.355248 (+ 3073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.355392 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.355644 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.356505 (+ 861us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.356643 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":264,"threads_started":1}
I20260619 02:57:18.359683 31314 tablet_service.cc:1467] Processing CreateTablet for tablet b979f8b10c4841eeb6c0616af0d79b67 (DEFAULT_TABLE table=test-table [id=cf44256982574ae1b4877352711bc583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:18.360321 31175 tablet_service.cc:1467] Processing CreateTablet for tablet b979f8b10c4841eeb6c0616af0d79b67 (DEFAULT_TABLE table=test-table [id=cf44256982574ae1b4877352711bc583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:18.361070 31314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b979f8b10c4841eeb6c0616af0d79b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:18.361438 31175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b979f8b10c4841eeb6c0616af0d79b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:18.364382 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.343733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43447 (local address 127.0.0.1:50728)
0619 02:57:18.354891 (+ 11158us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:18.354910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:18.354999 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:18.357973 (+ 2974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:18.357979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:18.358005 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:18.358162 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.358171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.360128 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.360133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.361853 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.361868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.363649 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.363655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.363859 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.363896 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:57:18.363960 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 02:57:18.364116 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11069,"thread_start_us":12323,"threads_started":1}
I20260619 02:57:18.365115 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.343989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.22.230.3:43447)
0619 02:57:18.357394 (+ 13405us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.357402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.357428 (+ 26us) server_negotiation.cc:407] Connection header received
0619 02:57:18.357527 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.357535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.357771 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.358322 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:18.358515 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.359954 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.362016 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.363464 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.364435 (+ 971us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.364635 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.364700 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.364838 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13256,"thread_start_us":13004,"threads_started":1}
I20260619 02:57:18.368541 31453 tablet_service.cc:1467] Processing CreateTablet for tablet b979f8b10c4841eeb6c0616af0d79b67 (DEFAULT_TABLE table=test-table [id=cf44256982574ae1b4877352711bc583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:18.369850 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b979f8b10c4841eeb6c0616af0d79b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:18.373412 31548 tablet_bootstrap.cc:492] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Bootstrap starting.
I20260619 02:57:18.375494 31549 tablet_bootstrap.cc:492] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Bootstrap starting.
I20260619 02:57:18.376737 31548 tablet_bootstrap.cc:654] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:18.377992 31548 log.cc:824] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:18.378821 31549 tablet_bootstrap.cc:654] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:18.379914 31549 log.cc:824] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:18.381006 31548 tablet_bootstrap.cc:492] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: No bootstrap required, opened a new log
I20260619 02:57:18.381295 31548 ts_tablet_manager.cc:1397] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 02:57:18.382356 31552 tablet_bootstrap.cc:492] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Bootstrap starting.
I20260619 02:57:18.382838 31549 tablet_bootstrap.cc:492] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: No bootstrap required, opened a new log
I20260619 02:57:18.383172 31549 ts_tablet_manager.cc:1397] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:57:18.385069 31552 tablet_bootstrap.cc:654] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:18.385984 31552 log.cc:824] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:18.388270 31552 tablet_bootstrap.cc:492] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: No bootstrap required, opened a new log
I20260619 02:57:18.388511 31552 ts_tablet_manager.cc:1397] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 02:57:18.391616 31548 raft_consensus.cc:348] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.392021 31548 raft_consensus.cc:374] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:18.392155 31548 raft_consensus.cc:729] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Initialized, Role: FOLLOWER
I20260619 02:57:18.392751 31548 consensus_queue.cc:260] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.393252 31549 raft_consensus.cc:348] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.393679 31549 raft_consensus.cc:374] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:18.393774 31549 raft_consensus.cc:729] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Initialized, Role: FOLLOWER
I20260619 02:57:18.394302 31549 consensus_queue.cc:260] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.396880 31548 ts_tablet_manager.cc:1428] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 02:57:18.398128 31549 ts_tablet_manager.cc:1428] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260619 02:57:18.398099 31552 raft_consensus.cc:348] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.398464 31552 raft_consensus.cc:374] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:18.398586 31552 raft_consensus.cc:729] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5246855b934e5fbedbc99c6f5ac904, State: Initialized, Role: FOLLOWER
I20260619 02:57:18.400058 31552 consensus_queue.cc:260] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.401376 31552 ts_tablet_manager.cc:1428] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260619 02:57:18.450178 31524 tablet.cc:2369] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:57:18.451939 31246 tablet.cc:2369] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:57:18.454731 31385 tablet.cc:2369] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:57:18.704704 31554 raft_consensus.cc:482] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:18.705024 31554 raft_consensus.cc:504] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.706790 31554 leader_election.cc:288] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:18.715464 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.707362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:52353 (local address 127.22.230.3:43447)
0619 02:57:18.707601 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.707610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.707969 (+ 359us) server_negotiation.cc:407] Connection header received
0619 02:57:18.708221 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.708226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.708299 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.708394 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:18.709121 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.710180 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.713047 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.714285 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.714419 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.714490 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.715103 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.715232 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:57:18.715464 31543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.707396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:41585 (local address 127.22.230.1:41743)
0619 02:57:18.707750 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.707756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.707807 (+ 51us) server_negotiation.cc:407] Connection header received
0619 02:57:18.708020 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.708028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.708109 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.708197 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:18.708855 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.710027 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.713047 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.714260 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.714419 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.714490 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.715103 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.715228 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260619 02:57:18.715739 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.707011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43447 (local address 127.22.230.2:52353)
0619 02:57:18.707886 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:18.707903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:18.708071 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:18.708429 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:18.708434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:18.708499 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:18.708964 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.708975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.710299 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.710306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.712866 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.712881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.714538 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.714544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.714848 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.714978 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 02:57:18.715064 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 02:57:18.715264 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":335,"thread_start_us":604,"threads_started":1}
I20260619 02:57:18.715794 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.706911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41743 (local address 127.22.230.2:41585)
0619 02:57:18.707690 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:18.707713 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:18.707848 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:18.708242 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:18.708246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:18.708267 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:18.708707 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.708717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.710187 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.710190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.712866 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:18.712881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.714537 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.714544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:18.714844 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.714970 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 02:57:18.715070 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 02:57:18.715264 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":247,"thread_start_us":378,"threads_started":1}
I20260619 02:57:18.716651 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979f8b10c4841eeb6c0616af0d79b67" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904" is_pre_election: true
I20260619 02:57:18.716658 31195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979f8b10c4841eeb6c0616af0d79b67" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db706169f8d4423abb7b8b5b52703c95" is_pre_election: true
I20260619 02:57:18.717099 31478 raft_consensus.cc:2454] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 0.
I20260619 02:57:18.717125 31195 raft_consensus.cc:2454] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 0.
I20260619 02:57:18.717664 31267 leader_election.cc:302] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5246855b934e5fbedbc99c6f5ac904, 9f048aa6ee9e402884c63d21a526dfbd; no voters:
I20260619 02:57:18.718022 31554 raft_consensus.cc:2781] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:18.718179 31554 raft_consensus.cc:482] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:18.718281 31554 raft_consensus.cc:3037] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:18.720350 31554 raft_consensus.cc:504] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.720860 31554 leader_election.cc:288] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 election: Requested vote from peers db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:18.721235 31195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979f8b10c4841eeb6c0616af0d79b67" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db706169f8d4423abb7b8b5b52703c95"
I20260619 02:57:18.721287 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979f8b10c4841eeb6c0616af0d79b67" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904"
I20260619 02:57:18.721417 31195 raft_consensus.cc:3037] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:18.721447 31478 raft_consensus.cc:3037] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:18.723537 31478 raft_consensus.cc:2454] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 1.
I20260619 02:57:18.723555 31195 raft_consensus.cc:2454] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 1.
I20260619 02:57:18.723881 31267 leader_election.cc:302] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5246855b934e5fbedbc99c6f5ac904, 9f048aa6ee9e402884c63d21a526dfbd; no voters:
I20260619 02:57:18.724133 31554 raft_consensus.cc:2781] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:18.724498 31554 raft_consensus.cc:686] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 LEADER]: Becoming Leader. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Running, Role: LEADER
I20260619 02:57:18.725003 31554 consensus_queue.cc:237] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.729633 31037 catalog_manager.cc:5495] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2). New cstate: current_term: 1 leader_uuid: "9f048aa6ee9e402884c63d21a526dfbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } health_report { overall_health: UNKNOWN } } }
I20260619 02:57:18.802664 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:18.797553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.22.230.2:42749)
0619 02:57:18.797770 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:18.797775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:18.797791 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:18.797898 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:18.797902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:18.797976 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:18.798089 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:18.798379 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.799074 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.800347 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:18.800752 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:18.800853 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:18.801494 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:18.802035 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:18.802134 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:18.802253 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 02:57:18.802311 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:57:18.810221 31478 raft_consensus.cc:1261] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Refusing update from remote peer 9f048aa6ee9e402884c63d21a526dfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:18.810220 31195 raft_consensus.cc:1261] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Refusing update from remote peer 9f048aa6ee9e402884c63d21a526dfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:18.811225 31559 consensus_queue.cc:1035] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:18.811537 31554 consensus_queue.cc:1035] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:18.823350 31564 mvcc.cc:205] Tried to move back new op lower bound from 7298407787757174784 to 7298407787421736960. Current Snapshot: MvccSnapshot[applied={T|T < 7298407787757174784}]
I20260619 02:57:18.823704 31562 mvcc.cc:205] Tried to move back new op lower bound from 7298407787757174784 to 7298407787421736960. Current Snapshot: MvccSnapshot[applied={T|T < 7298407787757174784}]
I20260619 02:57:18.825412 31563 mvcc.cc:205] Tried to move back new op lower bound from 7298407787757174784 to 7298407787421736960. Current Snapshot: MvccSnapshot[applied={T|T < 7298407787757174784}]
I20260619 02:57:18.836234 31038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:18.836494 31038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:18.839787 31038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Sending DeleteTablet for 3 replicas of tablet b979f8b10c4841eeb6c0616af0d79b67
I20260619 02:57:18.840823 31314 tablet_service.cc:1514] Processing DeleteTablet for tablet b979f8b10c4841eeb6c0616af0d79b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:18 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:52238
I20260619 02:57:18.840824 31175 tablet_service.cc:1514] Processing DeleteTablet for tablet b979f8b10c4841eeb6c0616af0d79b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:18 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:48056
I20260619 02:57:18.841084 31453 tablet_service.cc:1514] Processing DeleteTablet for tablet b979f8b10c4841eeb6c0616af0d79b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:18 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50728
I20260619 02:57:18.842263 31038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:57:18.843065 31038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:18.851868 31452 tablet_service.cc:1467] Processing CreateTablet for tablet a72a3ae391e748a6a3340f4ef449e7f9 (DEFAULT_TABLE table=test-table [id=25adb88076f9465292bf728bfd847603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:18.852586 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72a3ae391e748a6a3340f4ef449e7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:18.855014 31571 tablet_replica.cc:330] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: stopping tablet replica
I20260619 02:57:18.855353 31314 tablet_service.cc:1467] Processing CreateTablet for tablet a72a3ae391e748a6a3340f4ef449e7f9 (DEFAULT_TABLE table=test-table [id=25adb88076f9465292bf728bfd847603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:18.855460 31571 raft_consensus.cc:2229] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:18.855677 31572 tablet_replica.cc:330] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: stopping tablet replica
I20260619 02:57:18.855782 31314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72a3ae391e748a6a3340f4ef449e7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:18.856103 31572 raft_consensus.cc:2229] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:18.856125 31571 raft_consensus.cc:2258] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:18.856124 31174 tablet_service.cc:1467] Processing CreateTablet for tablet a72a3ae391e748a6a3340f4ef449e7f9 (DEFAULT_TABLE table=test-table [id=25adb88076f9465292bf728bfd847603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:18.856593 31572 raft_consensus.cc:2258] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:18.856806 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72a3ae391e748a6a3340f4ef449e7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:18.857304 31573 tablet_replica.cc:330] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: stopping tablet replica
I20260619 02:57:18.857653 31573 raft_consensus.cc:2229] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:18.858034 31573 raft_consensus.cc:2258] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:18.860031 31573 ts_tablet_manager.cc:1905] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:18.860462 31571 ts_tablet_manager.cc:1905] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:18.862663 31548 tablet_bootstrap.cc:492] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Bootstrap starting.
I20260619 02:57:18.864763 31548 tablet_bootstrap.cc:654] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:18.867117 31573 ts_tablet_manager.cc:1918] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:18.867340 31573 log.cc:1196] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/b979f8b10c4841eeb6c0616af0d79b67
I20260619 02:57:18.867923 31573 ts_tablet_manager.cc:1939] T b979f8b10c4841eeb6c0616af0d79b67 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting consensus metadata
I20260619 02:57:18.869740 31024 catalog_manager.cc:4841] TS 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447): tablet b979f8b10c4841eeb6c0616af0d79b67 (table test-table [id=cf44256982574ae1b4877352711bc583]) successfully deleted
I20260619 02:57:18.870975 31552 tablet_bootstrap.cc:492] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Bootstrap starting.
I20260619 02:57:18.872550 31552 tablet_bootstrap.cc:654] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:18.872684 31548 tablet_bootstrap.cc:492] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: No bootstrap required, opened a new log
I20260619 02:57:18.872826 31548 ts_tablet_manager.cc:1397] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260619 02:57:18.873670 31548 raft_consensus.cc:348] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.873888 31548 raft_consensus.cc:374] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:18.873993 31548 raft_consensus.cc:729] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Initialized, Role: FOLLOWER
I20260619 02:57:18.874301 31548 consensus_queue.cc:260] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.875072 31548 ts_tablet_manager.cc:1428] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 02:57:18.875401 31571 ts_tablet_manager.cc:1918] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:18.875917 31572 ts_tablet_manager.cc:1905] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:18.877084 31549 tablet_bootstrap.cc:492] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Bootstrap starting.
I20260619 02:57:18.877254 31571 log.cc:1196] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/b979f8b10c4841eeb6c0616af0d79b67
I20260619 02:57:18.877801 31571 ts_tablet_manager.cc:1939] T b979f8b10c4841eeb6c0616af0d79b67 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting consensus metadata
I20260619 02:57:18.878679 31549 tablet_bootstrap.cc:654] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:18.880722 31023 catalog_manager.cc:4841] TS 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749): tablet b979f8b10c4841eeb6c0616af0d79b67 (table test-table [id=cf44256982574ae1b4877352711bc583]) successfully deleted
I20260619 02:57:18.881246 31552 tablet_bootstrap.cc:492] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: No bootstrap required, opened a new log
I20260619 02:57:18.881433 31552 ts_tablet_manager.cc:1397] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 02:57:18.882233 31552 raft_consensus.cc:348] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.883884 31552 raft_consensus.cc:374] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:18.884040 31552 raft_consensus.cc:729] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5246855b934e5fbedbc99c6f5ac904, State: Initialized, Role: FOLLOWER
I20260619 02:57:18.884339 31552 consensus_queue.cc:260] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.887223 31572 ts_tablet_manager.cc:1918] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:18.886904 31552 ts_tablet_manager.cc:1428] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 02:57:18.888023 31572 log.cc:1196] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/b979f8b10c4841eeb6c0616af0d79b67
I20260619 02:57:18.888710 31549 tablet_bootstrap.cc:492] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: No bootstrap required, opened a new log
I20260619 02:57:18.888906 31572 ts_tablet_manager.cc:1939] T b979f8b10c4841eeb6c0616af0d79b67 P db706169f8d4423abb7b8b5b52703c95: Deleting consensus metadata
I20260619 02:57:18.888903 31549 ts_tablet_manager.cc:1397] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260619 02:57:18.889832 31549 raft_consensus.cc:348] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.890040 31549 raft_consensus.cc:374] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:18.890146 31549 raft_consensus.cc:729] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Initialized, Role: FOLLOWER
I20260619 02:57:18.890393 31549 consensus_queue.cc:260] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:18.890712 31025 catalog_manager.cc:4841] TS db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743): tablet b979f8b10c4841eeb6c0616af0d79b67 (table test-table [id=cf44256982574ae1b4877352711bc583]) successfully deleted
I20260619 02:57:18.891069 31549 ts_tablet_manager.cc:1428] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:57:19.047891 31555 raft_consensus.cc:482] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:19.048175 31555 raft_consensus.cc:504] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.049970 31555 leader_election.cc:288] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:19.058089 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.050837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:44435 (local address 127.22.230.2:42749)
0619 02:57:19.051063 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.051071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.051833 (+ 762us) server_negotiation.cc:407] Connection header received
0619 02:57:19.052047 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.052054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.052112 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.052220 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:19.052862 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.054054 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.056272 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.057084 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.057278 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.057352 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.057650 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.057774 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:57:19.058135 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.050311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:42749 (local address 127.22.230.1:44435)
0619 02:57:19.051726 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.051740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.051883 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.052246 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.052250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.052277 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:19.052667 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.052698 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.054202 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.054207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.056136 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.056144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.057308 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.057314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.057469 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.057517 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.057628 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.057761 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":74,"thread_start_us":1143,"threads_started":1}
I20260619 02:57:19.058270 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.050527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:53855 (local address 127.22.230.3:43447)
0619 02:57:19.050750 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.050756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.051315 (+ 559us) server_negotiation.cc:407] Connection header received
0619 02:57:19.051627 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.051634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.051723 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.051787 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:19.052721 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.053465 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.055837 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.057087 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.057245 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.057323 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.057872 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.058010 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:57:19.058965 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.050229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43447 (local address 127.22.230.1:53855)
0619 02:57:19.051204 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.051224 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.051435 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.052011 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.052016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.052043 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:19.052511 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.052525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.053606 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.053612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.055676 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.055689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.057471 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.057477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.057697 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.057728 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.058435 (+ 707us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.058592 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":35,"thread_start_us":836,"threads_started":1}
I20260619 02:57:19.059345 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72a3ae391e748a6a3340f4ef449e7f9" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f048aa6ee9e402884c63d21a526dfbd" is_pre_election: true
I20260619 02:57:19.059602 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72a3ae391e748a6a3340f4ef449e7f9" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904" is_pre_election: true
I20260619 02:57:19.059780 31339 raft_consensus.cc:2454] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 0.
I20260619 02:57:19.059887 31478 raft_consensus.cc:2454] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 0.
I20260619 02:57:19.060380 31127 leader_election.cc:302] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d5246855b934e5fbedbc99c6f5ac904, 9f048aa6ee9e402884c63d21a526dfbd, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:19.060813 31555 raft_consensus.cc:2781] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:19.060951 31555 raft_consensus.cc:482] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:19.061070 31555 raft_consensus.cc:3037] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.063287 31555 raft_consensus.cc:504] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.063959 31555 leader_election.cc:288] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 election: Requested vote from peers 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:19.064735 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72a3ae391e748a6a3340f4ef449e7f9" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904"
I20260619 02:57:19.064725 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72a3ae391e748a6a3340f4ef449e7f9" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f048aa6ee9e402884c63d21a526dfbd"
I20260619 02:57:19.064919 31339 raft_consensus.cc:3037] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.064914 31478 raft_consensus.cc:3037] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.067032 31478 raft_consensus.cc:2454] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 1.
I20260619 02:57:19.067425 31127 leader_election.cc:302] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5246855b934e5fbedbc99c6f5ac904, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:19.067694 31555 raft_consensus.cc:2781] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:19.068190 31555 raft_consensus.cc:686] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 1 LEADER]: Becoming Leader. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Running, Role: LEADER
I20260619 02:57:19.068295 31339 raft_consensus.cc:2454] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 1.
I20260619 02:57:19.068605 31555 consensus_queue.cc:237] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.071964 31038 catalog_manager.cc:5495] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 reported cstate change: term changed from 0 to 1, leader changed from <none> to db706169f8d4423abb7b8b5b52703c95 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "db706169f8d4423abb7b8b5b52703c95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } health_report { overall_health: UNKNOWN } } }
Jun 19 02:57:19 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837835, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.62@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:19.150481 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.136952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.22.230.62:46437)
0619 02:57:19.137907 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.137912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.137937 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:19.137972 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.137975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.138021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.138130 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:19.139276 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.140044 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.141299 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.141685 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.141816 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.145276 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:19.145281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:19.145284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:19.145309 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:19.147883 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:19.148551 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:19.148559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:19.148563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:19.148678 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:19.149023 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:19.149028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:19.149031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:19.149285 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:19.149434 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.149803 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.150046 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":815,"thread_start_us":629,"threads_started":1}
I20260619 02:57:19.150554 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.136927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:46437 (local address 127.0.0.1:55822)
0619 02:57:19.137595 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.137624 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.137797 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.138156 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.138160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.138887 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:19.139121 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.139129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.140167 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.140174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.141161 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.141171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.141853 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.141859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.142064 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.142736 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:19.142753 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:19.145059 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:19.148020 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:19.148034 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:19.148041 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:19.148441 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:19.148777 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:19.148784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:19.148788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:19.148916 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:19.149412 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:19.149420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:19.149574 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.149740 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.150148 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":156,"thread_start_us":362,"threads_started":1}
I20260619 02:57:19.156072 31038 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:55822:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 02:57:19.157019 31038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:19.163587 31174 tablet_service.cc:1467] Processing CreateTablet for tablet 2c04e93ae7544e3e80cfaedacf563cf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=26421d1f1d7c49ab8ac09d138819639b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:57:19.164215 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c04e93ae7544e3e80cfaedacf563cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.164157 31452 tablet_service.cc:1467] Processing CreateTablet for tablet 2c04e93ae7544e3e80cfaedacf563cf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=26421d1f1d7c49ab8ac09d138819639b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:57:19.164615 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c04e93ae7544e3e80cfaedacf563cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.167078 31314 tablet_service.cc:1467] Processing CreateTablet for tablet 2c04e93ae7544e3e80cfaedacf563cf4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=26421d1f1d7c49ab8ac09d138819639b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 02:57:19.167559 31314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c04e93ae7544e3e80cfaedacf563cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.171653 31552 tablet_bootstrap.cc:492] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: Bootstrap starting.
I20260619 02:57:19.172871 31549 tablet_bootstrap.cc:492] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: Bootstrap starting.
I20260619 02:57:19.173319 31552 tablet_bootstrap.cc:654] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.174543 31549 tablet_bootstrap.cc:654] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.177491 31548 tablet_bootstrap.cc:492] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: Bootstrap starting.
I20260619 02:57:19.179283 31548 tablet_bootstrap.cc:654] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.184981 31549 tablet_bootstrap.cc:492] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: No bootstrap required, opened a new log
I20260619 02:57:19.185171 31549 ts_tablet_manager.cc:1397] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260619 02:57:19.185743 31549 raft_consensus.cc:348] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.185946 31549 raft_consensus.cc:374] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.186022 31549 raft_consensus.cc:729] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.186250 31549 consensus_queue.cc:260] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.186858 31549 ts_tablet_manager.cc:1428] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 02:57:19.187621 31555 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.187872 31555 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.188923 31552 tablet_bootstrap.cc:492] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: No bootstrap required, opened a new log
I20260619 02:57:19.189133 31552 ts_tablet_manager.cc:1397] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20260619 02:57:19.189929 31552 raft_consensus.cc:348] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.190243 31552 raft_consensus.cc:374] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.190377 31552 raft_consensus.cc:729] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5246855b934e5fbedbc99c6f5ac904, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.190738 31552 consensus_queue.cc:260] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.190876 31548 tablet_bootstrap.cc:492] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: No bootstrap required, opened a new log
I20260619 02:57:19.191373 31552 ts_tablet_manager.cc:1428] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 02:57:19.191215 31556 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.191612 31556 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.191874 31548 ts_tablet_manager.cc:1397] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.002s
I20260619 02:57:19.192610 31548 raft_consensus.cc:348] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.192852 31548 raft_consensus.cc:374] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.193109 31548 raft_consensus.cc:729] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.193431 31548 consensus_queue.cc:260] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.194813 31559 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.195086 31559 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.195647 31548 ts_tablet_manager.cc:1428] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 02:57:19.300194 31556 raft_consensus.cc:482] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:19.300426 31556 raft_consensus.cc:504] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.301924 31556 leader_election.cc:288] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743), 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749)
I20260619 02:57:19.308840 31545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.302375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:48883 (local address 127.22.230.2:42749)
0619 02:57:19.302610 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.302620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.303677 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 02:57:19.303851 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.303860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.303913 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.304007 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:19.304527 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.305269 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.307155 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.307862 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.308039 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.308124 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.308405 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.308545 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:57:19.308971 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.302344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:33913 (local address 127.22.230.1:41743)
0619 02:57:19.303432 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.303439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.303459 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:57:19.303527 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.303533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.303603 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.303719 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:19.304282 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.305169 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.307148 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.307950 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.308179 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.308264 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.308525 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.308677 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":810,"threads_started":1}
I20260619 02:57:19.308992 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.302304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:42749 (local address 127.22.230.3:48883)
0619 02:57:19.303581 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.303593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.303726 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.304020 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.304028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.304054 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:19.304371 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.304380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.305397 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.305402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.307009 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.307020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.308071 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.308078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.308249 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.308271 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.308383 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.308501 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":30,"thread_start_us":1074,"threads_started":1}
I20260619 02:57:19.309269 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.302266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41743 (local address 127.22.230.3:33913)
0619 02:57:19.303160 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.303181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.303321 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.303749 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.303754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.303779 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:19.304101 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.304112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.305291 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.305299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.307003 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.307015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.308179 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.308185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.308386 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.308408 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.308502 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.308709 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":77,"thread_start_us":550,"threads_started":1}
I20260619 02:57:19.309921 31195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c04e93ae7544e3e80cfaedacf563cf4" candidate_uuid: "1d5246855b934e5fbedbc99c6f5ac904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db706169f8d4423abb7b8b5b52703c95" is_pre_election: true
I20260619 02:57:19.309998 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c04e93ae7544e3e80cfaedacf563cf4" candidate_uuid: "1d5246855b934e5fbedbc99c6f5ac904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f048aa6ee9e402884c63d21a526dfbd" is_pre_election: true
I20260619 02:57:19.310179 31195 raft_consensus.cc:2454] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d5246855b934e5fbedbc99c6f5ac904 in term 0.
I20260619 02:57:19.310273 31339 raft_consensus.cc:2454] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d5246855b934e5fbedbc99c6f5ac904 in term 0.
I20260619 02:57:19.310710 31407 leader_election.cc:302] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d5246855b934e5fbedbc99c6f5ac904, 9f048aa6ee9e402884c63d21a526dfbd, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:19.311144 31556 raft_consensus.cc:2781] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:19.311302 31556 raft_consensus.cc:482] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:19.311409 31556 raft_consensus.cc:3037] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.313311 31556 raft_consensus.cc:504] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.313952 31556 leader_election.cc:288] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [CANDIDATE]: Term 1 election: Requested vote from peers db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743), 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749)
I20260619 02:57:19.314342 31195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c04e93ae7544e3e80cfaedacf563cf4" candidate_uuid: "1d5246855b934e5fbedbc99c6f5ac904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db706169f8d4423abb7b8b5b52703c95"
I20260619 02:57:19.314364 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c04e93ae7544e3e80cfaedacf563cf4" candidate_uuid: "1d5246855b934e5fbedbc99c6f5ac904" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f048aa6ee9e402884c63d21a526dfbd"
I20260619 02:57:19.314543 31195 raft_consensus.cc:3037] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.314612 31339 raft_consensus.cc:3037] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.317493 31195 raft_consensus.cc:2454] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d5246855b934e5fbedbc99c6f5ac904 in term 1.
I20260619 02:57:19.317471 31339 raft_consensus.cc:2454] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d5246855b934e5fbedbc99c6f5ac904 in term 1.
I20260619 02:57:19.317874 31405 leader_election.cc:302] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5246855b934e5fbedbc99c6f5ac904, 9f048aa6ee9e402884c63d21a526dfbd; no voters:
I20260619 02:57:19.318135 31556 raft_consensus.cc:2781] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:19.318498 31556 raft_consensus.cc:686] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 LEADER]: Becoming Leader. State: Replica: 1d5246855b934e5fbedbc99c6f5ac904, State: Running, Role: LEADER
I20260619 02:57:19.319061 31556 consensus_queue.cc:237] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.320847 31594 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: TxnStatusTablet state changed. Reason: New leader 1d5246855b934e5fbedbc99c6f5ac904. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.321282 31594 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:57:19.321992 31596 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 02:57:19.322224 31596 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 02:57:19.322497 31037 catalog_manager.cc:5495] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3). New cstate: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } health_report { overall_health: HEALTHY } } }
I20260619 02:57:19.372335 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.366207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43447 (local address 127.0.0.1:50738)
0619 02:57:19.366423 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.366440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.366536 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.367097 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.367100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.367109 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:57:19.367247 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.367255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.368757 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.368761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.369824 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.369835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.370740 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.370744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.370871 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.370918 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.371871 (+ 953us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.371890 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.371958 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.372124 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 02:57:19.372402 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.366293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.22.230.3:43447)
0619 02:57:19.366483 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.366489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.366509 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:57:19.366780 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.366786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.366907 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.367085 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:19.367366 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.368622 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.369940 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.370618 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.370740 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.371085 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.371763 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.371904 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.371997 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.372096 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 02:57:19.376343 31195 raft_consensus.cc:1261] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Refusing update from remote peer 1d5246855b934e5fbedbc99c6f5ac904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:19.376513 31339 raft_consensus.cc:1261] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Refusing update from remote peer 1d5246855b934e5fbedbc99c6f5ac904: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:19.377113 31594 consensus_queue.cc:1035] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:19.377400 31556 consensus_queue.cc:1035] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:19.383432 31555 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: TxnStatusTablet state changed. Reason: New leader 1d5246855b934e5fbedbc99c6f5ac904. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.383803 31555 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.388010 31559 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: TxnStatusTablet state changed. Reason: New leader 1d5246855b934e5fbedbc99c6f5ac904. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.388302 31559 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.397207 31559 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.397464 31559 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 9f048aa6ee9e402884c63d21a526dfbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.397455 31555 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.397711 31555 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P db706169f8d4423abb7b8b5b52703c95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 02:57:19.405740 31594 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.405983 31594 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:57:19.406201 31594 tablet_replica.cc:439] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d5246855b934e5fbedbc99c6f5ac904" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } } }
I20260619 02:57:19.406440 31594 tablet_replica.cc:442] T 2c04e93ae7544e3e80cfaedacf563cf4 P 1d5246855b934e5fbedbc99c6f5ac904: This TxnStatusTablet replica's current role is: LEADER
I20260619 02:57:19.418001 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.411721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.22.230.1:41743)
0619 02:57:19.411892 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.411901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.412311 (+ 410us) server_negotiation.cc:407] Connection header received
0619 02:57:19.412543 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.412546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.412627 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.412774 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:19.413237 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.414310 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.415982 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.416385 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.416520 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.416927 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.417438 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.417597 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.417651 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.417727 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:57:19.432927 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.427181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43447 (local address 127.22.230.1:50057)
0619 02:57:19.427759 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.427774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.427880 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.428211 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.428214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.428223 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:57:19.428468 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.428475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.429835 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.429838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.430879 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.430889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.431925 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.431929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.432039 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.432084 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.432487 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.432494 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.432548 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.432634 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":341,"threads_started":1}
I20260619 02:57:19.433004 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.427271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:50057 (local address 127.22.230.3:43447)
0619 02:57:19.427419 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.427423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.427859 (+ 436us) server_negotiation.cc:407] Connection header received
0619 02:57:19.427996 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.428002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.428102 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.428196 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:19.428617 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.429709 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.431004 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.431766 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.431880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.432236 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.432397 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.432482 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.432598 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.432670 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:57:19.443048 31478 raft_consensus.cc:1261] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Refusing update from remote peer db706169f8d4423abb7b8b5b52703c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:19.443136 31339 raft_consensus.cc:1261] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Refusing update from remote peer db706169f8d4423abb7b8b5b52703c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:19.443801 31579 consensus_queue.cc:1035] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:19.444231 31555 consensus_queue.cc:1035] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:19.482450 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.476211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:59991 (local address 127.22.230.1:41743)
0619 02:57:19.476402 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.476409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.477081 (+ 672us) server_negotiation.cc:407] Connection header received
0619 02:57:19.477287 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.477292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.477366 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.477454 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:19.478016 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.478901 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.480749 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.481167 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.481311 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.481714 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.481818 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.481912 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.482060 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.482133 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:57:19.482583 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.476269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41743 (local address 127.22.230.3:59991)
0619 02:57:19.476985 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:19.477001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:19.477150 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:19.477474 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:19.477478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:19.477489 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 02:57:19.477860 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.477868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.479077 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.479085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.480608 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:19.480620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.481347 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.481354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:19.481493 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.481546 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.481914 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.481926 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 02:57:19.482042 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:57:19.482176 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":371,"threads_started":1}
I20260619 02:57:19.525447 31038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:19.525719 31038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:19.529084 31038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Sending DeleteTablet for 3 replicas of tablet a72a3ae391e748a6a3340f4ef449e7f9
I20260619 02:57:19.529848 31174 tablet_service.cc:1514] Processing DeleteTablet for tablet a72a3ae391e748a6a3340f4ef449e7f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:19 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:48056
I20260619 02:57:19.530197 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet a72a3ae391e748a6a3340f4ef449e7f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:19 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50728
I20260619 02:57:19.530474 31314 tablet_service.cc:1514] Processing DeleteTablet for tablet a72a3ae391e748a6a3340f4ef449e7f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:19 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:52238
I20260619 02:57:19.539459 31622 tablet_replica.cc:330] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: stopping tablet replica
I20260619 02:57:19.539666 31622 raft_consensus.cc:2229] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:19.539928 31622 raft_consensus.cc:2258] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:19.545608 31620 tablet_replica.cc:330] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: stopping tablet replica
I20260619 02:57:19.546487 31622 ts_tablet_manager.cc:1905] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:19.546581 31621 tablet_replica.cc:330] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: stopping tablet replica
I20260619 02:57:19.546814 31621 raft_consensus.cc:2229] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:19.546829 31620 raft_consensus.cc:2229] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:19.547134 31621 raft_consensus.cc:2258] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:19.547526 31620 raft_consensus.cc:2258] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:19.548317 31621 ts_tablet_manager.cc:1905] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:19.549294 31620 ts_tablet_manager.cc:1905] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:19.555959 31621 ts_tablet_manager.cc:1918] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:57:19.556097 31621 log.cc:1196] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/a72a3ae391e748a6a3340f4ef449e7f9
I20260619 02:57:19.556586 31621 ts_tablet_manager.cc:1939] T a72a3ae391e748a6a3340f4ef449e7f9 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting consensus metadata
I20260619 02:57:19.557689 31024 catalog_manager.cc:4841] TS 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447): tablet a72a3ae391e748a6a3340f4ef449e7f9 (table test-table [id=25adb88076f9465292bf728bfd847603]) successfully deleted
I20260619 02:57:19.561270 31622 ts_tablet_manager.cc:1918] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:57:19.563057 31620 ts_tablet_manager.cc:1918] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:57:19.563289 31622 log.cc:1196] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/a72a3ae391e748a6a3340f4ef449e7f9
I20260619 02:57:19.563789 31622 ts_tablet_manager.cc:1939] T a72a3ae391e748a6a3340f4ef449e7f9 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting consensus metadata
I20260619 02:57:19.564554 31620 log.cc:1196] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/a72a3ae391e748a6a3340f4ef449e7f9
I20260619 02:57:19.565233 31023 catalog_manager.cc:4841] TS 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749): tablet a72a3ae391e748a6a3340f4ef449e7f9 (table test-table [id=25adb88076f9465292bf728bfd847603]) successfully deleted
I20260619 02:57:19.565941 31620 ts_tablet_manager.cc:1939] T a72a3ae391e748a6a3340f4ef449e7f9 P db706169f8d4423abb7b8b5b52703c95: Deleting consensus metadata
I20260619 02:57:19.567229 31025 catalog_manager.cc:4841] TS db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743): tablet a72a3ae391e748a6a3340f4ef449e7f9 (table test-table [id=25adb88076f9465292bf728bfd847603]) successfully deleted
Jun 19 02:57:19 dist-test-slave-2gms krb5kdc[30993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 02:57:19 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:19 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:19.634362 31587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:19.621384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.22.230.62:46437)
0619 02:57:19.621590 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.621599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.621982 (+ 383us) server_negotiation.cc:407] Connection header received
0619 02:57:19.622167 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.622175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.622262 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.622374 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:19.623950 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.625136 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.627057 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.627715 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.627845 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.634090 (+ 6245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55834: BlockingRecv error: recv got EOF from 127.0.0.1:55834 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 02:57:19 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM
I20260619 02:57:19.657541 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.639290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.22.230.62:46437)
0619 02:57:19.640365 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.640372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.640391 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:57:19.640444 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.640448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.640532 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.640633 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:19.642224 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.643318 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.645135 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.645817 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.645948 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.650936 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:19.650944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:19.650950 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:19.650994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:19.654517 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:19.655392 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:19.655398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:19.655401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:19.655547 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:19.655964 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:19.655969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:19.655973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:19.656300 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:19.656561 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.656931 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.657165 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 02:57:19.661626 31038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:57:19.662350 31038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:19.670042 31174 tablet_service.cc:1467] Processing CreateTablet for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 (DEFAULT_TABLE table=test-table [id=a8a08cce90784b66b95bf64b31757b7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:19.670274 31452 tablet_service.cc:1467] Processing CreateTablet for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 (DEFAULT_TABLE table=test-table [id=a8a08cce90784b66b95bf64b31757b7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:19.670603 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.670830 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.675654 31314 tablet_service.cc:1467] Processing CreateTablet for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 (DEFAULT_TABLE table=test-table [id=a8a08cce90784b66b95bf64b31757b7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:19.676082 31314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.678186 31552 tablet_bootstrap.cc:492] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Bootstrap starting.
I20260619 02:57:19.679889 31552 tablet_bootstrap.cc:654] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.689673 31548 tablet_bootstrap.cc:492] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Bootstrap starting.
I20260619 02:57:19.691366 31548 tablet_bootstrap.cc:654] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.722903 31548 tablet_bootstrap.cc:492] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: No bootstrap required, opened a new log
I20260619 02:57:19.723160 31548 ts_tablet_manager.cc:1397] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20260619 02:57:19.724025 31548 raft_consensus.cc:348] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.724416 31548 raft_consensus.cc:374] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.724611 31548 raft_consensus.cc:729] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.724977 31548 consensus_queue.cc:260] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.729282 31548 ts_tablet_manager.cc:1428] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 02:57:19.736855 31552 tablet_bootstrap.cc:492] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: No bootstrap required, opened a new log
I20260619 02:57:19.737059 31552 ts_tablet_manager.cc:1397] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent bootstrapping tablet: real 0.059s user 0.018s sys 0.039s
I20260619 02:57:19.738170 31552 raft_consensus.cc:348] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.738550 31552 raft_consensus.cc:374] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.738713 31552 raft_consensus.cc:729] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5246855b934e5fbedbc99c6f5ac904, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.739039 31552 consensus_queue.cc:260] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.742538 31552 ts_tablet_manager.cc:1428] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260619 02:57:19.743084 31637 tablet_bootstrap.cc:492] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Bootstrap starting.
I20260619 02:57:19.744925 31637 tablet_bootstrap.cc:654] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.760064 31637 tablet_bootstrap.cc:492] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: No bootstrap required, opened a new log
I20260619 02:57:19.760295 31637 ts_tablet_manager.cc:1397] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260619 02:57:19.761261 31637 raft_consensus.cc:348] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.761570 31637 raft_consensus.cc:374] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.761719 31637 raft_consensus.cc:729] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.762040 31637 consensus_queue.cc:260] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.762702 31637 ts_tablet_manager.cc:1428] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 02:57:19.822872 31559 raft_consensus.cc:482] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:19.823101 31559 raft_consensus.cc:504] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.823707 31559 leader_election.cc:288] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:19.824240 31195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d49e8fcb91f4c4a8cd6927e4c14d1a4" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db706169f8d4423abb7b8b5b52703c95" is_pre_election: true
I20260619 02:57:19.824273 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d49e8fcb91f4c4a8cd6927e4c14d1a4" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904" is_pre_election: true
I20260619 02:57:19.824486 31195 raft_consensus.cc:2454] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 0.
I20260619 02:57:19.824535 31478 raft_consensus.cc:2454] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 0.
I20260619 02:57:19.825011 31268 leader_election.cc:302] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f048aa6ee9e402884c63d21a526dfbd, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:19.825398 31559 raft_consensus.cc:2781] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:19.825510 31559 raft_consensus.cc:482] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:19.825613 31559 raft_consensus.cc:3037] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.828537 31559 raft_consensus.cc:504] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.829144 31559 leader_election.cc:288] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 election: Requested vote from peers db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:19.829602 31195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d49e8fcb91f4c4a8cd6927e4c14d1a4" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db706169f8d4423abb7b8b5b52703c95"
I20260619 02:57:19.829592 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d49e8fcb91f4c4a8cd6927e4c14d1a4" candidate_uuid: "9f048aa6ee9e402884c63d21a526dfbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904"
I20260619 02:57:19.829769 31478 raft_consensus.cc:3037] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.829804 31195 raft_consensus.cc:3037] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:19.831993 31195 raft_consensus.cc:2454] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 1.
I20260619 02:57:19.832401 31268 leader_election.cc:302] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f048aa6ee9e402884c63d21a526dfbd, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:19.832643 31559 raft_consensus.cc:2781] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:19.832643 31478 raft_consensus.cc:2454] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f048aa6ee9e402884c63d21a526dfbd in term 1.
I20260619 02:57:19.833436 31559 raft_consensus.cc:686] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 LEADER]: Becoming Leader. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Running, Role: LEADER
I20260619 02:57:19.833703 31559 consensus_queue.cc:237] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.836529 31038 catalog_manager.cc:5495] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2). New cstate: current_term: 1 leader_uuid: "9f048aa6ee9e402884c63d21a526dfbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } health_report { overall_health: UNKNOWN } } }
I20260619 02:57:19.860081 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:19.846258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.22.230.2:42749)
0619 02:57:19.855065 (+ 8807us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:19.855070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:19.855090 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:57:19.855145 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:19.855148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:19.855208 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:19.855291 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:19.855765 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.856526 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.858376 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:19.858801 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:19.858889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:19.859351 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:19.859449 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:19.859511 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:19.859679 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 02:57:19.859734 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8697,"thread_start_us":8538,"threads_started":1}
I20260619 02:57:19.862716 31195 raft_consensus.cc:1261] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Refusing update from remote peer 9f048aa6ee9e402884c63d21a526dfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:19.862716 31478 raft_consensus.cc:1261] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Refusing update from remote peer 9f048aa6ee9e402884c63d21a526dfbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:19.863334 31639 consensus_queue.cc:1035] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:19.863963 31639 consensus_queue.cc:1035] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:19.877715 31037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:19.877964 31037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:19.880950 31037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Sending DeleteTablet for 3 replicas of tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4
I20260619 02:57:19.881687 31174 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:19 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:48056
I20260619 02:57:19.881968 31620 tablet_replica.cc:330] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: stopping tablet replica
I20260619 02:57:19.881943 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:19 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50728
I20260619 02:57:19.882164 31620 raft_consensus.cc:2229] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:19.882206 31621 tablet_replica.cc:330] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: stopping tablet replica
I20260619 02:57:19.882367 31620 raft_consensus.cc:2258] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:19.882392 31621 raft_consensus.cc:2229] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:19.882634 31621 raft_consensus.cc:2258] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:19.883615 31621 ts_tablet_manager.cc:1905] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:19.883899 31620 ts_tablet_manager.cc:1905] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:19.886857 31314 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:19 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:52238
I20260619 02:57:19.887135 31622 tablet_replica.cc:330] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: stopping tablet replica
I20260619 02:57:19.887324 31622 raft_consensus.cc:2229] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:19.887607 31622 raft_consensus.cc:2258] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:19.888257 31037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 02:57:19.888798 31622 ts_tablet_manager.cc:1905] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 02:57:19.889101 31037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:19.891556 31620 ts_tablet_manager.cc:1918] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:19.891703 31620 log.cc:1196] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/3d49e8fcb91f4c4a8cd6927e4c14d1a4
I20260619 02:57:19.892120 31620 ts_tablet_manager.cc:1939] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P db706169f8d4423abb7b8b5b52703c95: Deleting consensus metadata
I20260619 02:57:19.893236 31025 catalog_manager.cc:4841] TS db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743): tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 (table test-table [id=a8a08cce90784b66b95bf64b31757b7d]) successfully deleted
I20260619 02:57:19.896306 31174 tablet_service.cc:1467] Processing CreateTablet for tablet 02fb6daabcfb4cdfb98d0d96746aa812 (DEFAULT_TABLE table=test-table [id=d9cad20f3d844279be9f8d7bc95bed66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:19.896737 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02fb6daabcfb4cdfb98d0d96746aa812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.896634 31452 tablet_service.cc:1467] Processing CreateTablet for tablet 02fb6daabcfb4cdfb98d0d96746aa812 (DEFAULT_TABLE table=test-table [id=d9cad20f3d844279be9f8d7bc95bed66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:19.897050 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02fb6daabcfb4cdfb98d0d96746aa812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.899631 31314 tablet_service.cc:1467] Processing CreateTablet for tablet 02fb6daabcfb4cdfb98d0d96746aa812 (DEFAULT_TABLE table=test-table [id=d9cad20f3d844279be9f8d7bc95bed66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:19.899997 31314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02fb6daabcfb4cdfb98d0d96746aa812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:19.903970 31637 tablet_bootstrap.cc:492] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Bootstrap starting.
I20260619 02:57:19.905404 31637 tablet_bootstrap.cc:654] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.908330 31548 tablet_bootstrap.cc:492] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Bootstrap starting.
I20260619 02:57:19.909873 31548 tablet_bootstrap.cc:654] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.909953 31622 ts_tablet_manager.cc:1918] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:19.896629 31621 ts_tablet_manager.cc:1918] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:19.910249 31621 log.cc:1196] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/3d49e8fcb91f4c4a8cd6927e4c14d1a4
I20260619 02:57:19.910692 31622 log.cc:1196] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/3d49e8fcb91f4c4a8cd6927e4c14d1a4
I20260619 02:57:19.910766 31621 ts_tablet_manager.cc:1939] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting consensus metadata
I20260619 02:57:19.911937 31024 catalog_manager.cc:4841] TS 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447): tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 (table test-table [id=a8a08cce90784b66b95bf64b31757b7d]) successfully deleted
I20260619 02:57:19.912283 31622 ts_tablet_manager.cc:1939] T 3d49e8fcb91f4c4a8cd6927e4c14d1a4 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting consensus metadata
I20260619 02:57:19.912781 31552 tablet_bootstrap.cc:492] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Bootstrap starting.
I20260619 02:57:19.913388 31023 catalog_manager.cc:4841] TS 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749): tablet 3d49e8fcb91f4c4a8cd6927e4c14d1a4 (table test-table [id=a8a08cce90784b66b95bf64b31757b7d]) successfully deleted
I20260619 02:57:19.914533 31552 tablet_bootstrap.cc:654] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:19.915243 31637 tablet_bootstrap.cc:492] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: No bootstrap required, opened a new log
I20260619 02:57:19.915454 31637 ts_tablet_manager.cc:1397] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260619 02:57:19.916532 31637 raft_consensus.cc:348] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.916882 31637 raft_consensus.cc:374] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.916994 31637 raft_consensus.cc:729] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.918094 31637 consensus_queue.cc:260] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.919050 31637 ts_tablet_manager.cc:1428] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:19.923159 31548 tablet_bootstrap.cc:492] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: No bootstrap required, opened a new log
I20260619 02:57:19.923339 31548 ts_tablet_manager.cc:1397] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260619 02:57:19.923779 31552 tablet_bootstrap.cc:492] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: No bootstrap required, opened a new log
I20260619 02:57:19.923990 31552 ts_tablet_manager.cc:1397] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20260619 02:57:19.924090 31548 raft_consensus.cc:348] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.924275 31548 raft_consensus.cc:374] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.924329 31548 raft_consensus.cc:729] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f048aa6ee9e402884c63d21a526dfbd, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.924580 31548 consensus_queue.cc:260] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.925038 31548 ts_tablet_manager.cc:1428] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 02:57:19.924968 31552 raft_consensus.cc:348] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.925287 31552 raft_consensus.cc:374] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:19.925371 31552 raft_consensus.cc:729] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5246855b934e5fbedbc99c6f5ac904, State: Initialized, Role: FOLLOWER
I20260619 02:57:19.925643 31552 consensus_queue.cc:260] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:19.926162 31552 ts_tablet_manager.cc:1428] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 02:57:20.060694 31555 raft_consensus.cc:482] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:20.060925 31555 raft_consensus.cc:504] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:20.061710 31555 leader_election.cc:288] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:20.062317 31338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fb6daabcfb4cdfb98d0d96746aa812" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f048aa6ee9e402884c63d21a526dfbd" is_pre_election: true
I20260619 02:57:20.062585 31338 raft_consensus.cc:2454] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 0.
I20260619 02:57:20.062485 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fb6daabcfb4cdfb98d0d96746aa812" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904" is_pre_election: true
I20260619 02:57:20.062853 31478 raft_consensus.cc:2454] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 0.
I20260619 02:57:20.063053 31126 leader_election.cc:302] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f048aa6ee9e402884c63d21a526dfbd, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:20.063314 31555 raft_consensus.cc:2781] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:20.063416 31555 raft_consensus.cc:482] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:20.063493 31555 raft_consensus.cc:3037] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:20.066394 31555 raft_consensus.cc:504] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:20.067484 31338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fb6daabcfb4cdfb98d0d96746aa812" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f048aa6ee9e402884c63d21a526dfbd"
I20260619 02:57:20.067709 31338 raft_consensus.cc:3037] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:20.070622 31338 raft_consensus.cc:2454] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 1.
I20260619 02:57:20.070947 31555 leader_election.cc:288] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 election: Requested vote from peers 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749), 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447)
I20260619 02:57:20.070995 31126 leader_election.cc:302] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f048aa6ee9e402884c63d21a526dfbd, db706169f8d4423abb7b8b5b52703c95; no voters:
I20260619 02:57:20.071257 31555 raft_consensus.cc:2781] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:20.071689 31555 raft_consensus.cc:686] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 1 LEADER]: Becoming Leader. State: Replica: db706169f8d4423abb7b8b5b52703c95, State: Running, Role: LEADER
I20260619 02:57:20.071628 31478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02fb6daabcfb4cdfb98d0d96746aa812" candidate_uuid: "db706169f8d4423abb7b8b5b52703c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5246855b934e5fbedbc99c6f5ac904"
I20260619 02:57:20.071827 31478 raft_consensus.cc:3037] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:20.072021 31555 consensus_queue.cc:237] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } }
I20260619 02:57:20.074532 31478 raft_consensus.cc:2454] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db706169f8d4423abb7b8b5b52703c95 in term 1.
I20260619 02:57:20.074785 31038 catalog_manager.cc:5495] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 reported cstate change: term changed from 0 to 1, leader changed from <none> to db706169f8d4423abb7b8b5b52703c95 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "db706169f8d4423abb7b8b5b52703c95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db706169f8d4423abb7b8b5b52703c95" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 } health_report { overall_health: UNKNOWN } } }
I20260619 02:57:20.104437 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:20.099808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.22.230.1:41743)
0619 02:57:20.100307 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:20.100312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:20.100326 (+ 14us) server_negotiation.cc:407] Connection header received
0619 02:57:20.100374 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:20.100377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:20.100431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:20.100510 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:20.100835 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.101570 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.102928 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.103365 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.103456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:20.103829 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:20.103932 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:20.103996 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:20.104134 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 02:57:20.104189 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260619 02:57:20.112774 31338 raft_consensus.cc:1261] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Refusing update from remote peer db706169f8d4423abb7b8b5b52703c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:20.112805 31478 raft_consensus.cc:1261] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Refusing update from remote peer db706169f8d4423abb7b8b5b52703c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:20.113277 31649 consensus_queue.cc:1035] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f048aa6ee9e402884c63d21a526dfbd" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:20.113596 31555 consensus_queue.cc:1035] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5246855b934e5fbedbc99c6f5ac904" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:20.159343 31038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:20.159554 31038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:20.161903 31038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d24fc5772484d1686a0843c6906fb58: Sending DeleteTablet for 3 replicas of tablet 02fb6daabcfb4cdfb98d0d96746aa812
I20260619 02:57:20.162541 31174 tablet_service.cc:1514] Processing DeleteTablet for tablet 02fb6daabcfb4cdfb98d0d96746aa812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:20 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:48056
I20260619 02:57:20.162624 31314 tablet_service.cc:1514] Processing DeleteTablet for tablet 02fb6daabcfb4cdfb98d0d96746aa812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:20 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:52238
I20260619 02:57:20.162861 31622 tablet_replica.cc:330] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: stopping tablet replica
I20260619 02:57:20.162861 31620 tablet_replica.cc:330] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: stopping tablet replica
I20260619 02:57:20.163034 31622 raft_consensus.cc:2229] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:20.163034 31620 raft_consensus.cc:2229] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:20.163221 31622 raft_consensus.cc:2258] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:20.163340 31620 raft_consensus.cc:2258] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:20.163619 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet 02fb6daabcfb4cdfb98d0d96746aa812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:20 UTC) from {username='oryx', principal='oryx/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50728
I20260619 02:57:20.163873 31621 tablet_replica.cc:330] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: stopping tablet replica
I20260619 02:57:20.164065 31621 raft_consensus.cc:2229] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:20.164264 31621 raft_consensus.cc:2258] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:20.164270 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31109
I20260619 02:57:20.164623 31622 ts_tablet_manager.cc:1905] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:20.165619 31620 ts_tablet_manager.cc:1905] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:20.165920 31621 ts_tablet_manager.cc:1905] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:20.170415 31621 ts_tablet_manager.cc:1918] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:57:20.170500 31621 log.cc:1196] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/02fb6daabcfb4cdfb98d0d96746aa812
I20260619 02:57:20.171344 31621 ts_tablet_manager.cc:1939] T 02fb6daabcfb4cdfb98d0d96746aa812 P 1d5246855b934e5fbedbc99c6f5ac904: Deleting consensus metadata
I20260619 02:57:20.171871 31620 ts_tablet_manager.cc:1918] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:57:20.171995 31620 log.cc:1196] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/02fb6daabcfb4cdfb98d0d96746aa812
I20260619 02:57:20.172341 31622 ts_tablet_manager.cc:1918] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 02:57:20.172367 31024 catalog_manager.cc:4841] TS 1d5246855b934e5fbedbc99c6f5ac904 (127.22.230.3:43447): tablet 02fb6daabcfb4cdfb98d0d96746aa812 (table test-table [id=d9cad20f3d844279be9f8d7bc95bed66]) successfully deleted
I20260619 02:57:20.172465 31620 ts_tablet_manager.cc:1939] T 02fb6daabcfb4cdfb98d0d96746aa812 P db706169f8d4423abb7b8b5b52703c95: Deleting consensus metadata
I20260619 02:57:20.172513 31622 log.cc:1196] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/02fb6daabcfb4cdfb98d0d96746aa812
I20260619 02:57:20.172924 31622 ts_tablet_manager.cc:1939] T 02fb6daabcfb4cdfb98d0d96746aa812 P 9f048aa6ee9e402884c63d21a526dfbd: Deleting consensus metadata
I20260619 02:57:20.173533 31025 catalog_manager.cc:4841] TS db706169f8d4423abb7b8b5b52703c95 (127.22.230.1:41743): tablet 02fb6daabcfb4cdfb98d0d96746aa812 (table test-table [id=d9cad20f3d844279be9f8d7bc95bed66]) successfully deleted
I20260619 02:57:20.173840 31023 catalog_manager.cc:4841] TS 9f048aa6ee9e402884c63d21a526dfbd (127.22.230.2:42749): tablet 02fb6daabcfb4cdfb98d0d96746aa812 (table test-table [id=d9cad20f3d844279be9f8d7bc95bed66]) successfully deleted
Jun 19 02:57:20 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.1@KRBTEST.COM
I20260619 02:57:20.184629 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:20.171552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.22.230.1:41743)
0619 02:57:20.171780 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:20.171787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:20.171806 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:57:20.171863 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:20.171868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:20.172006 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:20.172205 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:20.174025 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.174799 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.175931 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.176289 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.176377 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:20.179916 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:20.179928 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:20.179943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:20.179970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:20.182399 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:20.183130 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:20.183136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:20.183138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:20.183251 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:20.183524 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:20.183527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:20.183528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:20.183832 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:20.184053 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:20.184235 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 02:57:20.184399 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:57:20.185246 31240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:20.310875 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31109
I20260619 02:57:20.334381 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31249
Jun 19 02:57:20 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.2@KRBTEST.COM
I20260619 02:57:20.347733 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:20.335177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.22.230.2:42749)
0619 02:57:20.335349 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:20.335354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:20.335372 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:57:20.335472 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:20.335476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:20.335541 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:20.335621 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:20.336744 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.337414 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.338534 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.338998 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.339089 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:20.342994 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:20.343005 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:20.343020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:20.343050 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:20.345503 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:20.346032 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:20.346038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:20.346040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:20.346141 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:20.346432 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:20.346435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:20.346437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:20.346810 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:20.347032 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:20.347266 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 02:57:20.347404 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:57:20.348345 31379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:20.468243 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31249
I20260619 02:57:20.490167 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31388
Jun 19 02:57:20 dist-test-slave-2gms krb5kdc[30993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.230.3@KRBTEST.COM
I20260619 02:57:20.504326 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:20.490932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.22.230.3:43447)
0619 02:57:20.491401 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:20.491406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:20.491422 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:20.491473 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:20.491477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:20.491536 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:20.491623 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:20.492784 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.493462 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.494553 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:20.495000 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:20.495089 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:20.499144 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:20.499159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:20.499174 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:20.499206 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:20.501999 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:20.502620 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:20.502632 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:20.502638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:20.502807 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:20.503124 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:20.503127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:20.503129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:20.503433 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:20.503626 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:20.503914 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 02:57:20.504075 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260619 02:57:20.504940 31518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:20.629631 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31388
I20260619 02:57:20.652019 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31006
I20260619 02:57:20.653141 31078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:20.764266 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31006
2026-06-19T02:57:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6902 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:57:20 dist-test-slave-2gms krb5kdc[31669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:57:20 dist-test-slave-2gms krb5kdc[31669](info): set up 2 sockets
Jun 19 02:57:20 dist-test-slave-2gms krb5kdc[31669](info): commencing operation
krb5kdc: starting...
W20260619 02:57:21.851557 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:21 dist-test-slave-2gms krb5kdc[31669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:57:22.076042 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:43501
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:43501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:22.383896 31682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:22.384166 31682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:22.384334 31682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:22.391222 31682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:22.391314 31682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:22.391410 31682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:22.391513 31682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:22.391579 31682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:22.400918 31682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:43501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:22.402107 31682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:22.403347 31682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:22.414840 31688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:22.414790 31687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:22.414875 31690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:22.415441 31682 server_base.cc:1034] running on GCE node
I20260619 02:57:22.416153 31682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:22.417397 31682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:22.418634 31682 hybrid_clock.cc:648] HybridClock initialized: now 1781837842418551 us; error 34 us; skew 500 ppm
I20260619 02:57:22.418926 31682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T02:57:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1715 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:57:22 dist-test-slave-2gms krb5kdc[31696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:57:22 dist-test-slave-2gms krb5kdc[31696](info): set up 2 sockets
Jun 19 02:57:22 dist-test-slave-2gms krb5kdc[31696](info): commencing operation
krb5kdc: starting...
W20260619 02:57:23.567322 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:23 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:57:23.771606 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:42821
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:24.076504 31709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:24.076748 31709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:24.076916 31709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:24.083762 31709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:24.083870 31709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:24.083976 31709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:24.084043 31709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:24.084098 31709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:24.093111 31709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:24.094344 31709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:24.095582 31709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:24.106904 31715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:24.106966 31717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:24.106930 31714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:24.107569 31709 server_base.cc:1034] running on GCE node
I20260619 02:57:24.108359 31709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:24.109647 31709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:24.110862 31709 hybrid_clock.cc:648] HybridClock initialized: now 1781837844110826 us; error 48 us; skew 500 ppm
I20260619 02:57:24.111271 31709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:24 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837844, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:24.117249 31709 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:57:24.118664 31709 webserver.cc:460] Webserver started at http://127.22.230.62:41583/ using document root <none> and password file <none>
I20260619 02:57:24.119297 31709 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:24.119429 31709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:24.119720 31709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:24.122407 31709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "58e8e7bdab714a3f99eec0ac7631efd5"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "f8b46253ee115dc6f1569a81dc7f6f82"
server_key_iv: "940be45e1685ac132b366688d28712a6"
server_key_version: "encryptionkey@0"
I20260619 02:57:24.123196 31709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "58e8e7bdab714a3f99eec0ac7631efd5"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "f8b46253ee115dc6f1569a81dc7f6f82"
server_key_iv: "940be45e1685ac132b366688d28712a6"
server_key_version: "encryptionkey@0"
I20260619 02:57:24.127955 31709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 02:57:24.131145 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:24.132320 31709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:24.132481 31709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "58e8e7bdab714a3f99eec0ac7631efd5"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "f8b46253ee115dc6f1569a81dc7f6f82"
server_key_iv: "940be45e1685ac132b366688d28712a6"
server_key_version: "encryptionkey@0"
I20260619 02:57:24.132655 31709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:24.147073 31709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:24.150746 31709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:24.151018 31709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:24.168818 31709 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:42821
I20260619 02:57:24.168840 31775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:42821 every 8 connection(s)
I20260619 02:57:24.170141 31709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:24.173563 31776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:24.178732 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31709
I20260619 02:57:24.178929 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:24.179692 23448 external_mini_cluster.cc:1428] Setting key d29e4879c43b77ecdb7cb0abf65545a8
I20260619 02:57:24.184142 31776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Bootstrap starting.
I20260619 02:57:24.187597 31776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:24.188798 31776 log.cc:824] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:24.191474 31776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: No bootstrap required, opened a new log
Jun 19 02:57:24 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:24.199071 31776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:24.199446 31776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:24.199582 31776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e8e7bdab714a3f99eec0ac7631efd5, State: Initialized, Role: FOLLOWER
I20260619 02:57:24.200215 31776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:24.200464 31776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:24.200662 31776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:24.200850 31776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:24.201200 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:24.181618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.22.230.62:42821)
0619 02:57:24.182374 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:24.182385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:24.182423 (+ 38us) server_negotiation.cc:407] Connection header received
0619 02:57:24.184019 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:24.184039 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:24.184457 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:24.184743 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:24.186428 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:24.187860 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:24.189584 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:24.190240 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:24.190424 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:24.194149 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:24.194157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:24.194172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:24.194206 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:24.197155 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:24.197792 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:24.197799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:24.197804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:24.197975 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:24.198329 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:24.198333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:24.198338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:24.198893 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:24.199179 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:24.199604 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 02:57:24.200032 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":358,"threads_started":1}
I20260619 02:57:24.203630 31776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:24.204120 31776 leader_election.cc:302] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e8e7bdab714a3f99eec0ac7631efd5; no voters:
I20260619 02:57:24.204645 31776 leader_election.cc:288] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:24.204772 31781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:24.206054 31781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 LEADER]: Becoming Leader. State: Replica: 58e8e7bdab714a3f99eec0ac7631efd5, State: Running, Role: LEADER
I20260619 02:57:24.206529 31781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:24.206696 31776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:24.209721 31783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e8e7bdab714a3f99eec0ac7631efd5. Latest consensus state: current_term: 1 leader_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } } }
I20260619 02:57:24.209748 31782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } } }
I20260619 02:57:24.210136 31782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:24.210139 31783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:24.211771 31791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:24.215911 31791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:24.223670 31791 catalog_manager.cc:1347] Generated new cluster ID: 54eecc2d362e4f7a8a7bea1333efdd41
I20260619 02:57:24.223769 31791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:24.250433 31791 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:24.251693 31791 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:24.258423 31791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Generated new TSK 0
I20260619 02:57:24.259315 31791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:57:24.318114 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42821
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:24.624032 31802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:24.624339 31802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:24.624459 31802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:24.631163 31802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:24.631253 31802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:24.631353 31802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:24.640626 31802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:24.641726 31802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:24.642928 31802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:24.655705 31807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:24.655750 31810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:24.655791 31808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:24.656515 31802 server_base.cc:1034] running on GCE node
I20260619 02:57:24.657037 31802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:24.657820 31802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:24.659031 31802 hybrid_clock.cc:648] HybridClock initialized: now 1781837844658981 us; error 58 us; skew 500 ppm
I20260619 02:57:24.659315 31802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:24 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837844, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:24.662214 31802 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:57:24.663545 31802 webserver.cc:460] Webserver started at http://127.22.230.1:37745/ using document root <none> and password file <none>
I20260619 02:57:24.664196 31802 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:24.664335 31802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:24.664636 31802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:24.667423 31802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "72ac2fa6328c4980a23b4befeeabe51e"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "671add2db006f79ed58323249b2e0cd2"
server_key_iv: "5ed43b6f049e56c32f31d6899d51c4cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:24.668184 31802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "72ac2fa6328c4980a23b4befeeabe51e"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "671add2db006f79ed58323249b2e0cd2"
server_key_iv: "5ed43b6f049e56c32f31d6899d51c4cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:24.672860 31802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 02:57:24.676087 31817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:24.677291 31802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:57:24.677452 31802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "72ac2fa6328c4980a23b4befeeabe51e"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "671add2db006f79ed58323249b2e0cd2"
server_key_iv: "5ed43b6f049e56c32f31d6899d51c4cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:24.677642 31802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:24.691596 31802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:24.695604 31802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:24.695866 31802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:24.696715 31802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:24.697908 31802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:24.698019 31802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:24.698156 31802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:24.698228 31802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:24.733784 31802 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:42933
I20260619 02:57:24.733829 31929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:42933 every 8 connection(s)
I20260619 02:57:24.735060 31802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:24.737684 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31802
I20260619 02:57:24.737893 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:24.738729 23448 external_mini_cluster.cc:1428] Setting key 4d30f7079a2cddb4ffa9090eb10426f8
Jun 19 02:57:24 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837844, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:24.754997 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:24.738078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:53183 (local address 127.22.230.62:42821)
0619 02:57:24.738517 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:24.738522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:24.739812 (+ 1290us) server_negotiation.cc:407] Connection header received
0619 02:57:24.741694 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:24.741699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:24.741779 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:24.741899 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:24.743445 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:24.744182 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:24.745717 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:24.746103 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:24.746234 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:24.749735 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:24.749739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:24.749743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:24.749771 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:24.752221 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:24.752880 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:24.752886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:24.752893 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:24.753000 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:24.753404 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:24.753409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:24.753412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:24.753664 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:24.753808 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:24.754413 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 02:57:24.754645 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":243,"threads_started":1}
I20260619 02:57:24.755846 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:24.738635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:53183)
0619 02:57:24.739410 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:24.739466 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:24.741439 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:24.742139 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:24.742148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:24.742731 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:24.743286 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:24.743302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:24.744337 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:24.744342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:24.745572 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:24.745583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:24.746318 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:24.746324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:24.746550 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:24.747314 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:24.747333 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:24.749537 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:24.752361 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:24.752369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:24.752383 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:24.752757 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:24.753101 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:24.753107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:24.753110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:24.753289 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:24.753784 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:24.753791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:24.754085 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 02:57:24.754318 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 02:57:24.754804 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":306,"threads_started":1}
I20260619 02:57:24.757321 31930 heartbeater.cc:347] Connected to a master server at 127.22.230.62:42821
I20260619 02:57:24.757637 31930 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:24.758493 31930 heartbeater.cc:511] Master 127.22.230.62:42821 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:57:24.760715 31741 ts_manager.cc:194] Registered new tserver with Master: 72ac2fa6328c4980a23b4befeeabe51e (127.22.230.1:42933)
I20260619 02:57:24.762012 31741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:53183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:57:24.789474 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42821
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:25.094177 31936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:25.094413 31936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:25.094502 31936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:25.101346 31936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:25.101438 31936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:25.101544 31936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:25.110931 31936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:25.112043 31936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:25.113238 31936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:25.125545 31944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:25.125613 31942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:25.125761 31941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:25.126510 31936 server_base.cc:1034] running on GCE node
I20260619 02:57:25.127090 31936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:25.127915 31936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:25.129096 31936 hybrid_clock.cc:648] HybridClock initialized: now 1781837845129050 us; error 53 us; skew 500 ppm
I20260619 02:57:25.129370 31936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:25 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837845, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:25.132266 31936 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:57:25.133504 31936 webserver.cc:460] Webserver started at http://127.22.230.2:38401/ using document root <none> and password file <none>
I20260619 02:57:25.134097 31936 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:25.134249 31936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:25.134486 31936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:25.137202 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "335e60482b2e4a77ac0fe0ee0c869061"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "7b6519f9dc68d0225b5d4ff356931f42"
server_key_iv: "aa33d6fcf7a6f889d0bb1e3fce39f9ef"
server_key_version: "encryptionkey@0"
I20260619 02:57:25.137898 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "335e60482b2e4a77ac0fe0ee0c869061"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "7b6519f9dc68d0225b5d4ff356931f42"
server_key_iv: "aa33d6fcf7a6f889d0bb1e3fce39f9ef"
server_key_version: "encryptionkey@0"
I20260619 02:57:25.142355 31936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 02:57:25.145588 31951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:25.146852 31936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:25.147019 31936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "335e60482b2e4a77ac0fe0ee0c869061"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "7b6519f9dc68d0225b5d4ff356931f42"
server_key_iv: "aa33d6fcf7a6f889d0bb1e3fce39f9ef"
server_key_version: "encryptionkey@0"
I20260619 02:57:25.147207 31936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:25.167573 31936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:25.171590 31936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:25.171885 31936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:25.172744 31936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:25.174012 31936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:25.174109 31936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:25.174252 31936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:25.174321 31936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:25.205739 31936 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:35441
I20260619 02:57:25.205799 32063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:35441 every 8 connection(s)
I20260619 02:57:25.207101 31936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:25.208612 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 31936
I20260619 02:57:25.208791 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:25.209276 23448 external_mini_cluster.cc:1428] Setting key 514f33d3f642fa08717765d97cb93568
Jun 19 02:57:25 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837845, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:25.227767 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:25.210070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:43863 (local address 127.22.230.62:42821)
0619 02:57:25.210281 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:25.210287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:25.211935 (+ 1648us) server_negotiation.cc:407] Connection header received
0619 02:57:25.213863 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:25.213870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:25.213948 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:25.214049 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:25.215435 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.216535 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.218115 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.218834 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.218969 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:25.222421 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:25.222426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:25.222429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:25.222456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:25.224973 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.225629 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.225636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.225639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.225759 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.226156 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.226162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.226166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.226401 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:25.226540 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:25.227225 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 02:57:25.227433 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:57:25.228719 32066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:25.210483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:43863)
0619 02:57:25.211766 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:25.211836 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:25.213598 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:25.214218 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:25.214224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:25.214747 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:25.215277 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:25.215296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.216669 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.216673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:25.217957 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:25.217968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.219053 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.219063 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:25.219361 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:25.220055 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:25.220087 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:25.222224 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:25.225110 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:25.225120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:25.225135 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:25.225510 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:25.225862 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:25.225868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:25.225871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:25.226043 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:25.226515 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:25.226522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:25.226829 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 02:57:25.227132 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 02:57:25.227599 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":705,"threads_started":1}
I20260619 02:57:25.230010 32064 heartbeater.cc:347] Connected to a master server at 127.22.230.62:42821
I20260619 02:57:25.230350 32064 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:25.231215 32064 heartbeater.cc:511] Master 127.22.230.62:42821 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:57:25.232677 31741 ts_manager.cc:194] Registered new tserver with Master: 335e60482b2e4a77ac0fe0ee0c869061 (127.22.230.2:35441)
I20260619 02:57:25.233422 31741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:43863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:57:25.259474 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42821
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:25.566385 32069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:25.566656 32069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:25.566871 32069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:25.573602 32069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:25.573691 32069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:25.573863 32069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:25.583264 32069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:25.584508 32069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:25.585736 32069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:25.598204 32074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:25.598167 32075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:25.598232 32077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:25.598944 32069 server_base.cc:1034] running on GCE node
I20260619 02:57:25.599448 32069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:25.600230 32069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:25.601449 32069 hybrid_clock.cc:648] HybridClock initialized: now 1781837845601407 us; error 52 us; skew 500 ppm
I20260619 02:57:25.601747 32069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:25 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837845, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:25.604619 32069 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:57:25.605918 32069 webserver.cc:460] Webserver started at http://127.22.230.3:36005/ using document root <none> and password file <none>
I20260619 02:57:25.606552 32069 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:25.606719 32069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:25.607009 32069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:25.610285 32069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "b426586ae6644e6898e1dc97ef73e1ce"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "27316551b7e2fb7aab39527b7d670185"
server_key_iv: "dd75a754b50f55dd0919de7fb73c8807"
server_key_version: "encryptionkey@0"
I20260619 02:57:25.611228 32069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "b426586ae6644e6898e1dc97ef73e1ce"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "27316551b7e2fb7aab39527b7d670185"
server_key_iv: "dd75a754b50f55dd0919de7fb73c8807"
server_key_version: "encryptionkey@0"
I20260619 02:57:25.616768 32069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 02:57:25.620816 32084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:25.622118 32069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:25.622291 32069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "b426586ae6644e6898e1dc97ef73e1ce"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "27316551b7e2fb7aab39527b7d670185"
server_key_iv: "dd75a754b50f55dd0919de7fb73c8807"
server_key_version: "encryptionkey@0"
I20260619 02:57:25.622509 32069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:25.649487 32069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:25.653870 32069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:25.654203 32069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:25.655582 32069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:25.657755 32069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:25.657835 32069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:25.657922 32069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:25.657979 32069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:25.692029 32069 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:36713
I20260619 02:57:25.692071 32196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:36713 every 8 connection(s)
I20260619 02:57:25.693311 32069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:25.698909 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 32069
I20260619 02:57:25.699079 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:25.699496 23448 external_mini_cluster.cc:1428] Setting key 0d1b4f7b9dc8d15081137851574d2baf
Jun 19 02:57:25 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837845, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:25.713759 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:25.695795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:58983 (local address 127.22.230.62:42821)
0619 02:57:25.695982 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:25.695986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:25.697034 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 02:57:25.698942 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:25.698949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:25.699046 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:25.699171 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:25.700952 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.701940 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.703765 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.704216 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.704385 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:25.707916 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:25.707923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:25.707928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:25.707968 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:25.710670 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.711409 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.711415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.711419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.711551 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.712000 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.712006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.712010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.712281 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:25.712450 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:25.713193 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 02:57:25.713430 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:57:25.714898 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:25.696163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:58983)
0619 02:57:25.696855 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:25.696893 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:25.698658 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:25.699403 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:25.699411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:25.700069 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:25.700781 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:25.700801 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.702279 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.702285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:25.703632 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:25.703643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.704483 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.704493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:25.704759 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:25.705371 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:25.705393 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:25.707710 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:25.710805 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:25.710816 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:25.710835 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:25.711272 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:25.711687 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:25.711692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:25.711696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:25.711871 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:25.712420 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:25.712428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:25.712769 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 02:57:25.713058 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 02:57:25.713566 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":254,"threads_started":1}
I20260619 02:57:25.716091 32197 heartbeater.cc:347] Connected to a master server at 127.22.230.62:42821
I20260619 02:57:25.716331 32197 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:25.716949 32197 heartbeater.cc:511] Master 127.22.230.62:42821 requested a full tablet report, sending...
I20260619 02:57:25.718181 31741 ts_manager.cc:194] Registered new tserver with Master: b426586ae6644e6898e1dc97ef73e1ce (127.22.230.3:36713)
I20260619 02:57:25.719012 31741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:58983
I20260619 02:57:25.726326 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.1@KRBTEST.COM" created.
I20260619 02:57:25.764768 31930 heartbeater.cc:503] Master 127.22.230.62:42821 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab.
Entry for principal oryx/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab.
WARNING: no policy specified for oryx/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab.
Entry for principal oryx/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab.
WARNING: no policy specified for oryx/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab.
Entry for principal oryx/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab.
I20260619 02:57:25.850463 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31802
Jun 19 02:57:25 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:57:25.866721 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:25.851746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.22.230.1:42933)
0619 02:57:25.852453 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:25.852461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:25.852485 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:57:25.852540 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:25.852547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:25.852753 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:25.852923 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:25.854190 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.855300 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.856554 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.857114 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.857255 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:25.861118 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:25.861129 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:25.861143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:25.861176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:25.863933 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.864564 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.864579 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.864582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.864733 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.865121 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.865125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.865127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.865457 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:25.865759 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:25.866040 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 02:57:25.866297 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":364,"threads_started":1}
I20260619 02:57:25.868216 31925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:25.964176 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31802
I20260619 02:57:25.980070 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31936
Jun 19 02:57:25 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:57:25.994854 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:25.980881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.22.230.2:35441)
0619 02:57:25.981437 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:25.981445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:25.981470 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:25.981529 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:25.981536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:25.981737 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:25.981864 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:25.983012 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.983997 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.985125 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:25.985619 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:25.985708 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:25.989677 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:25.989688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:25.989701 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:25.989736 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:25.992250 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.992895 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.992902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.992904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.993017 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:25.993363 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:25.993370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:25.993373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:25.993679 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:25.993966 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:25.994257 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 02:57:25.994502 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":299,"threads_started":1}
I20260619 02:57:25.996016 32059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:26.077486 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31936
I20260619 02:57:26.093305 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32069
Jun 19 02:57:26 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:57:26.108690 32211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:26.094232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.22.230.3:36713)
0619 02:57:26.094817 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:26.094823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:26.094851 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:57:26.094909 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:26.094915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:26.095133 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:26.095275 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:26.096446 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:26.097432 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:26.098635 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:26.099162 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:26.099277 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:26.103572 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:26.103583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:26.103596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:26.103630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:26.106315 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:26.106983 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:26.106990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:26.106992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:26.107149 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:26.107450 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:26.107453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:26.107455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:26.107781 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:26.108066 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:26.108225 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 02:57:26.108416 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":256,"threads_started":1}
I20260619 02:57:26.110023 32192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:26.194430 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 32069
I20260619 02:57:26.209488 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 31709
I20260619 02:57:26.210677 31771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:26.307533 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 31709
I20260619 02:57:26.328923 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:42821
--webserver_interface=127.22.230.62
--webserver_port=41583
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:26.638648 32214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:26.638936 32214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:26.639104 32214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:26.645817 32214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:26.645902 32214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:26.645943 32214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:26.646018 32214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:26.646090 32214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:26.656188 32214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=41583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:26.657415 32214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:26.658998 32214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:26.670603 32220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:26.670689 32222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:26.670715 32219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:26.671608 32214 server_base.cc:1034] running on GCE node
I20260619 02:57:26.672308 32214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:26.673472 32214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:26.674747 32214 hybrid_clock.cc:648] HybridClock initialized: now 1781837846674720 us; error 28 us; skew 500 ppm
I20260619 02:57:26.675079 32214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:26 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837846, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:26.678051 32214 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:57:26.679445 32214 webserver.cc:460] Webserver started at http://127.22.230.62:41583/ using document root <none> and password file <none>
I20260619 02:57:26.680064 32214 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:26.680210 32214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:26.685251 32214 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 02:57:26.687883 32229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:26.689079 32214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:26.689251 32214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "58e8e7bdab714a3f99eec0ac7631efd5"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "f8b46253ee115dc6f1569a81dc7f6f82"
server_key_iv: "940be45e1685ac132b366688d28712a6"
server_key_version: "encryptionkey@0"
I20260619 02:57:26.690155 32214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:26.712194 32214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:26.716354 32214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:26.716606 32214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:26.733168 32214 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:42821
I20260619 02:57:26.733191 32280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:42821 every 8 connection(s)
I20260619 02:57:26.734606 32214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:26.739813 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 32214
I20260619 02:57:26.740800 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:42933
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:42821
--webserver_port=37745
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:57:26.741904 32281 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:57:26.746841 32281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Bootstrap starting.
I20260619 02:57:26.765767 32281 log.cc:824] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:26.774121 32281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:57:26.774691 32281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Bootstrap complete.
I20260619 02:57:26.783429 32281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:26.784163 32281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e8e7bdab714a3f99eec0ac7631efd5, State: Initialized, Role: FOLLOWER
I20260619 02:57:26.784749 32281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:26.784962 32281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:26.785096 32281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:26.785272 32281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:57:26.788970 32281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:26.789422 32281 leader_election.cc:302] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e8e7bdab714a3f99eec0ac7631efd5; no voters:
I20260619 02:57:26.790078 32281 leader_election.cc:288] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:57:26.790212 32285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:57:26.791585 32285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [term 2 LEADER]: Becoming Leader. State: Replica: 58e8e7bdab714a3f99eec0ac7631efd5, State: Running, Role: LEADER
I20260619 02:57:26.792142 32285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } }
I20260619 02:57:26.792289 32281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:26.794517 32287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e8e7bdab714a3f99eec0ac7631efd5. Latest consensus state: current_term: 2 leader_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } } }
I20260619 02:57:26.794859 32287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:26.795411 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e8e7bdab714a3f99eec0ac7631efd5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42821 } } }
I20260619 02:57:26.795715 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:26.806684 32293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:26.812140 32293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:26.813738 32293 catalog_manager.cc:1259] Loaded cluster ID: 54eecc2d362e4f7a8a7bea1333efdd41
I20260619 02:57:26.813885 32293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:26.817672 32293 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:26.819131 32293 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 58e8e7bdab714a3f99eec0ac7631efd5: Loaded TSK: 0
I20260619 02:57:26.820518 32293 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:57:27.065169 32283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:27.065462 32283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:27.065615 32283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:27.072715 32283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:27.072849 32283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:27.073030 32283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:27.082479 32283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.1.keytab
--principal=oryx/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:42933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=37745
--tserver_master_addrs=127.22.230.62:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:27.083750 32283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:27.085023 32283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:27.099083 32307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:27.099057 32308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:27.100095 32310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:27.100229 32283 server_base.cc:1034] running on GCE node
I20260619 02:57:27.100860 32283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:27.101857 32283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:27.103094 32283 hybrid_clock.cc:648] HybridClock initialized: now 1781837847103041 us; error 50 us; skew 500 ppm
I20260619 02:57:27.103410 32283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:27 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837847, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:27.106437 32283 init.cc:377] Logged in from keytab as oryx/127.22.230.1@KRBTEST.COM (short username oryx)
I20260619 02:57:27.108003 32283 webserver.cc:460] Webserver started at http://127.22.230.1:37745/ using document root <none> and password file <none>
I20260619 02:57:27.108666 32283 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:27.108805 32283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:27.114079 32283 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:57:27.116726 32317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:27.118114 32283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:27.118295 32283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "72ac2fa6328c4980a23b4befeeabe51e"
format_stamp: "Formatted at 2026-06-19 02:57:24 on dist-test-slave-2gms"
server_key: "671add2db006f79ed58323249b2e0cd2"
server_key_iv: "5ed43b6f049e56c32f31d6899d51c4cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:27.119241 32283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:27.139784 32283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:27.143795 32283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:27.144052 32283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:27.145056 32283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:27.146520 32283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:27.146656 32283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 02:57:27.146780 32283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:27.146878 32283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:27.178773 32283 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:42933
I20260619 02:57:27.178858 32429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:42933 every 8 connection(s)
I20260619 02:57:27.180166 32283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:27.184697 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 32283
I20260619 02:57:27.185417 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:35441
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:42821
--webserver_port=38401
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:57:27 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:27 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:27.591166 32434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:27.591590 32434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:27.591814 32434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:27.607568 32434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:27.607722 32434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:27.607973 32434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:27.628837 32434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.2.keytab
--principal=oryx/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:35441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=38401
--tserver_master_addrs=127.22.230.62:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:27.630553 32434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:27.632450 32434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:27.654592 32441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:27.654649 32440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:27.668393 32443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:27.668869 32434 server_base.cc:1034] running on GCE node
I20260619 02:57:27.669718 32434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:27.671078 32434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:27.672406 32434 hybrid_clock.cc:648] HybridClock initialized: now 1781837847672309 us; error 81 us; skew 500 ppm
I20260619 02:57:27.672855 32434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:27 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837847, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:27.678015 32434 init.cc:377] Logged in from keytab as oryx/127.22.230.2@KRBTEST.COM (short username oryx)
I20260619 02:57:27.680703 32434 webserver.cc:460] Webserver started at http://127.22.230.2:38401/ using document root <none> and password file <none>
I20260619 02:57:27.682035 32434 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:27.682245 32434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:27.692276 32434 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 02:57:27.697157 32450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:27.699375 32434 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 02:57:27.699606 32434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "335e60482b2e4a77ac0fe0ee0c869061"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "7b6519f9dc68d0225b5d4ff356931f42"
server_key_iv: "aa33d6fcf7a6f889d0bb1e3fce39f9ef"
server_key_version: "encryptionkey@0"
I20260619 02:57:27.701704 32434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:27.732091 32434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:27.739262 32434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:27.739732 32434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:27.753082 32434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:27.755513 32434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:27.755703 32434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 02:57:27.755882 32434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:27.756021 32434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:27.813498 32434 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:35441
I20260619 02:57:27.813545 32562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:35441 every 8 connection(s)
I20260619 02:57:27.816236 32434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:27.816958 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 32434
I20260619 02:57:27.817811 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:36713
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:42821
--webserver_port=36005
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:57:27 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:27 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:28.357087 32432 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.162s user 0.000s sys 0.013s
I20260619 02:57:28.358951 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:27.182812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:38029)
0619 02:57:27.183579 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:27.183619 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:27.185966 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:27.187406 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:27.187420 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:27.188192 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:27.188928 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:27.188953 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.190448 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.190456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:27.192561 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:27.192578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.193899 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.193909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:27.194212 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:27.194996 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:27.195036 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:28.357858 (+1162822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":308,"threads_started":1}
W20260619 02:57:28.358999 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:27.182792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:38029 (local address 127.22.230.62:42821)
0619 02:57:27.183594 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:27.183608 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:27.183704 (+ 96us) server_negotiation.cc:407] Connection header received
0619 02:57:27.186513 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:27.186525 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:27.186902 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:27.187264 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:27.189124 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.190291 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.192742 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.193639 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.193776 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.358225 (+1164449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:38029: BlockingRecv error: recv got EOF from 127.22.230.1:38029 (error 108)
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":350,"threads_started":1}
W20260619 02:57:28.360221 32430 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:42821 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.230.62:42821: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:28.370853 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:28.360899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:37513)
0619 02:57:28.361241 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:28.361256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:28.361417 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:28.361743 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:28.361747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:28.362098 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:28.362245 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:28.362253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.363305 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.363310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:28.364222 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:28.364232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.364914 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.364920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:28.365082 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.365716 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:28.365732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:28.370368 (+ 4636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252}
W20260619 02:57:28.370878 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:28.360977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:37513 (local address 127.22.230.62:42821)
0619 02:57:28.361132 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:28.361136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:28.361329 (+ 193us) server_negotiation.cc:407] Connection header received
0619 02:57:28.361548 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:28.361553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:28.361618 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:28.361721 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:28.362377 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.363164 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.364351 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.364734 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.364885 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.370628 (+ 5743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:37513: BlockingRecv error: recv got EOF from 127.22.230.1:37513 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:28.385185 32433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:28.372206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:47675 (local address 127.22.230.62:42821)
0619 02:57:28.372353 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:28.372359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:28.372375 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:28.372420 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:28.372425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:28.372513 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:28.372605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:28.373721 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.374847 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.377425 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.378084 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.378207 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.384973 (+ 6766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:47675: BlockingRecv error: recv got EOF from 127.22.230.1:47675 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:57:28.385193 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:28.371896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:47675)
0619 02:57:28.372134 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:28.372163 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:28.372303 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:28.372685 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:28.372693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:28.373253 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:28.373544 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:28.373559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.375020 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.375029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:28.377244 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:28.377264 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.378264 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.378274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:28.378611 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.379353 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:28.379386 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:28.384793 (+ 5407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 02:57:28.385710 32430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:57:28.418726 32564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:28.418956 32564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:28.419144 32564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:28.426013 32564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:28.426121 32564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:28.426297 32564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:28.435374 32564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/krb5kdc/oryx_127.22.230.3.keytab
--principal=oryx/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:36713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=36005
--tserver_master_addrs=127.22.230.62:42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:28.436563 32564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:28.437809 32564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:28.451602 32573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:28.451722 32576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:28.452536 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:28.453629 32564 server_base.cc:1034] running on GCE node
I20260619 02:57:28.454267 32564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:28.455340 32564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:28.456550 32564 hybrid_clock.cc:648] HybridClock initialized: now 1781837848456471 us; error 73 us; skew 500 ppm
I20260619 02:57:28.456840 32564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837848, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:28.459865 32564 init.cc:377] Logged in from keytab as oryx/127.22.230.3@KRBTEST.COM (short username oryx)
I20260619 02:57:28.461369 32564 webserver.cc:460] Webserver started at http://127.22.230.3:36005/ using document root <none> and password file <none>
I20260619 02:57:28.462008 32564 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:28.462133 32564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:28.467620 32564 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:57:28.470248 32583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:28.471758 32564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:57:28.471932 32564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "b426586ae6644e6898e1dc97ef73e1ce"
format_stamp: "Formatted at 2026-06-19 02:57:25 on dist-test-slave-2gms"
server_key: "27316551b7e2fb7aab39527b7d670185"
server_key_iv: "dd75a754b50f55dd0919de7fb73c8807"
server_key_version: "encryptionkey@0"
I20260619 02:57:28.472842 32564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:28.504949 32564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:28.509227 32564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:28.509491 32564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:28.510502 32564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:28.511895 32564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:28.512009 32564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:28.512136 32564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:28.512224 32564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:28.549839 32564 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:36713
I20260619 02:57:28.549887 32695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:36713 every 8 connection(s)
I20260619 02:57:28.551265 32564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:28.552565 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 32564
I20260619 02:57:28.570971 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:28.553877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.22.230.62:42821)
0619 02:57:28.554119 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:28.554128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:28.554584 (+ 456us) server_negotiation.cc:407] Connection header received
0619 02:57:28.554775 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:28.554786 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:28.554846 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:28.554945 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:28.556307 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.557634 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.559055 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.559944 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.560088 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.564150 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:28.564163 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:28.564187 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:28.564241 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:28.567859 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:28.568476 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:28.568488 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:28.568493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:28.568665 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:28.569028 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:28.569033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:28.569036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:28.569450 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:28.569745 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:28.570122 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 02:57:28.570509 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:28 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:29.083606 32566 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.210s user 0.001s sys 0.024s
W20260619 02:57:29.085122 32567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:27.838968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:51101 (local address 127.22.230.62:42821)
0619 02:57:27.855386 (+ 16418us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:27.855397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:27.856815 (+ 1418us) server_negotiation.cc:407] Connection header received
0619 02:57:27.860663 (+ 3848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:27.860674 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:27.860823 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:27.860944 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:27.864519 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.866206 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.870363 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.871320 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.871449 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.084698 (+1213249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:51101: BlockingRecv error: recv got EOF from 127.22.230.2:51101 (error 108)
Metrics: {"server-negotiator.queue_time_us":16222,"thread_start_us":16059,"threads_started":1}
I20260619 02:57:29.085165 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:27.838202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:51101)
0619 02:57:27.856252 (+ 18050us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:27.856341 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:27.860133 (+ 3792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:27.861815 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:27.861834 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:27.863020 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:27.864194 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:27.864227 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.866514 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.866524 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:27.870107 (+ 3583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:27.870148 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:27.871952 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:27.871961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:27.872363 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:27.873335 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:27.873378 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.084339 (+1210961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17581,"thread_start_us":17288,"threads_started":1}
W20260619 02:57:29.086112 32563 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:42821 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.230.62:42821: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:29.096798 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:29.086960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:43657)
0619 02:57:29.087189 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:29.087209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:29.087354 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:29.087670 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:29.087674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:29.088014 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:29.088176 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.088184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.089154 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.089159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.090113 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.090124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.090907 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.090913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.091089 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.091798 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:29.091863 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.096507 (+ 4644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 02:57:29.096881 32567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:29.087002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:43657 (local address 127.22.230.62:42821)
0619 02:57:29.087199 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:29.087207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:29.087286 (+ 79us) server_negotiation.cc:407] Connection header received
0619 02:57:29.087473 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:29.087478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:29.087547 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:29.087643 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:29.088306 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.089015 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.090256 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.090742 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.090879 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.096680 (+ 5801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:43657: BlockingRecv error: recv got EOF from 127.22.230.2:43657 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:29.108871 32567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:29.097981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:50953 (local address 127.22.230.62:42821)
0619 02:57:29.098121 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:29.098125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:29.098258 (+ 133us) server_negotiation.cc:407] Connection header received
0619 02:57:29.098454 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:29.098458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:29.098507 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:29.098622 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:29.099637 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.100307 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.103028 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.103638 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.103749 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.108683 (+ 4934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:50953: BlockingRecv error: recv got EOF from 127.22.230.2:50953 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 02:57:29.108947 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:29.097867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:50953)
0619 02:57:29.098159 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:29.098184 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:29.098325 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:29.098670 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:29.098677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:29.099206 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:29.099476 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.099491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.100453 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.100461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.102866 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.102886 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.103773 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.103779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.103981 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.104578 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:29.104599 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.108494 (+ 3895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149}
W20260619 02:57:29.109450 32563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 02:57:29.399848 32567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:29.386497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:52277 (local address 127.22.230.62:42821)
0619 02:57:29.386762 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:29.386772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:29.387081 (+ 309us) server_negotiation.cc:407] Connection header received
0619 02:57:29.387240 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:29.387246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:29.387339 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:29.387445 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:29.388372 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.389312 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.390551 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.390979 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.391132 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.399594 (+ 8462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:52277: BlockingRecv error: recv got EOF from 127.22.230.1:52277 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 02:57:29.399873 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:29.386335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:52277)
0619 02:57:29.386961 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:29.386999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:29.387106 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:29.387490 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:29.387496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:29.388035 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:29.388234 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.388244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.389440 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.389454 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.390394 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.390405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.391178 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.391184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.391357 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.395315 (+ 3958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:29.395331 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.399361 (+ 4030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":354,"threads_started":1}
W20260619 02:57:29.669344 32699 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.089s user 0.001s sys 0.012s
W20260619 02:57:29.670790 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:28.556630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:46913 (local address 127.22.230.62:42821)
0619 02:57:28.570978 (+ 14348us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:28.570985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:28.571004 (+ 19us) server_negotiation.cc:407] Connection header received
0619 02:57:28.571048 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:28.571054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:28.571148 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:28.571242 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:28.573963 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.574947 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.578369 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.578783 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.578869 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.670468 (+1091599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:46913: BlockingRecv error: recv got EOF from 127.22.230.3:46913 (error 108)
Metrics: {"server-negotiator.queue_time_us":14200,"thread_start_us":14111,"threads_started":1}
I20260619 02:57:29.671573 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:28.555964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:46913)
0619 02:57:28.564551 (+ 8587us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:28.564607 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:28.567141 (+ 2534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:28.571674 (+ 4533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:28.571693 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:28.572697 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:28.573740 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:28.573771 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.575129 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.575138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:28.578184 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:28.578206 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:28.579031 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:28.579042 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:28.579491 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:28.580266 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:28.580306 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.670206 (+1089900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8266,"thread_start_us":8007,"threads_started":1}
W20260619 02:57:29.672542 32696 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:42821 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.230.62:42821: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:29.681114 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:29.673186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:60501)
0619 02:57:29.673355 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:29.673375 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:29.673463 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:29.673745 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:29.673748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:29.674043 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:29.674209 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.674216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.675253 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.675256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.676366 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.676378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.677079 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.677084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.677199 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.677726 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:29.677742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.680927 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 02:57:29.681383 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:29.673196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:60501 (local address 127.22.230.62:42821)
0619 02:57:29.673417 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:29.673423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:29.673440 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:57:29.673581 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:29.673584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:29.673637 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:29.673709 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:29.674327 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.675144 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.676513 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.676920 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.677043 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.681113 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:60501: BlockingRecv error: recv got EOF from 127.22.230.3:60501 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:29 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:29.689463 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:29.681704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:53457)
0619 02:57:29.681864 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:29.681882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:29.682007 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:29.682313 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:29.682317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:29.682679 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:29.682815 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.682822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.683752 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.683758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.684940 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:29.684951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.685592 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.685597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:29.685783 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.686244 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:29.686259 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:29.689210 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:57:29.689616 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:29.681780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:53457 (local address 127.22.230.62:42821)
0619 02:57:29.681929 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:29.681935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:29.681951 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:57:29.682113 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:29.682118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:29.682193 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:29.682286 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:29.682936 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.683621 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.685078 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:29.685452 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:29.685582 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:29.689367 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:53457: BlockingRecv error: recv got EOF from 127.22.230.3:53457 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 02:57:29.689743 32696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 02:57:30 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:30 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:30.118219 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:30.109893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:37371)
0619 02:57:30.110389 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:30.110405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:30.110514 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:30.110880 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:30.110883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:30.111276 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:30.111411 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:30.111418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.112314 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.112318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:30.113240 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:30.113250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.113898 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.113907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:30.114041 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:30.114768 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:30.114789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:30.118005 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
W20260619 02:57:30.118364 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:30.109995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:37371 (local address 127.22.230.62:42821)
0619 02:57:30.110197 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:30.110206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:30.110484 (+ 278us) server_negotiation.cc:407] Connection header received
0619 02:57:30.110688 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:30.110692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:30.110762 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:30.110841 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:30.111523 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.112201 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.113348 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.113730 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.113815 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:30.118185 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:37371: BlockingRecv error: recv got EOF from 127.22.230.2:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:57:30 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:30 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:30.409581 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:30.401011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:56791)
0619 02:57:30.401452 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:30.401468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:30.401575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:30.401840 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:30.401843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:30.402339 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:30.402496 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:30.402505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.403494 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.403498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:30.404372 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:30.404382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.405132 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.405138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:30.405322 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:30.405927 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:30.405945 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:30.409218 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":236,"threads_started":1}
W20260619 02:57:30.409618 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:30.401077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:56791 (local address 127.22.230.62:42821)
0619 02:57:30.401264 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:30.401273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:30.401542 (+ 269us) server_negotiation.cc:407] Connection header received
0619 02:57:30.401667 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:30.401670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:30.401727 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:30.401803 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:30.402648 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.403378 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.404543 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.404972 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.405121 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:30.409392 (+ 4271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:56791: BlockingRecv error: recv got EOF from 127.22.230.1:56791 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 02:57:30 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:30 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:30.698704 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:30.690436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:34397)
0619 02:57:30.690940 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:30.690955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:30.691057 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:30.691313 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:30.691316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:30.691687 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:30.691815 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:30.691821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.692698 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.692701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:30.693749 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:30.693760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.694459 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.694464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:30.694627 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:30.695277 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:30.695293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:30.698329 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":301,"threads_started":1}
W20260619 02:57:30.698758 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:30.690484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:34397 (local address 127.22.230.62:42821)
0619 02:57:30.690705 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:30.690714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:30.691025 (+ 311us) server_negotiation.cc:407] Connection header received
0619 02:57:30.691134 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:30.691137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:30.691203 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:30.691279 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:30.691935 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.692594 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.693870 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:30.694321 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:30.694409 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:30.698529 (+ 4120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:34397: BlockingRecv error: recv got EOF from 127.22.230.3:34397 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 02:57:31 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:31 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:31.128039 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:31.119099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:49771)
0619 02:57:31.119642 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:31.119657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:31.119777 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:31.120018 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:31.120021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:31.120368 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:31.120550 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:31.120558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.121482 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.121485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:31.122536 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:31.122548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.123213 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.123219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:31.123409 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:31.124073 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:31.124089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:31.127728 (+ 3639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":320,"threads_started":1}
W20260619 02:57:31.128118 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:31.119111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:49771 (local address 127.22.230.62:42821)
0619 02:57:31.119244 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:31.119248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:31.119726 (+ 478us) server_negotiation.cc:407] Connection header received
0619 02:57:31.119851 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:31.119854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:31.119906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:31.119979 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:31.120658 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.121366 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.122665 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.123038 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.123197 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:31.127919 (+ 4722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:49771: BlockingRecv error: recv got EOF from 127.22.230.2:49771 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 02:57:31 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:31 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:31.418432 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:31.410324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:44341)
0619 02:57:31.410851 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:31.410866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:31.410971 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:31.411232 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:31.411235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:31.411545 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:31.411694 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:31.411702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.412608 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.412611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:31.413499 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:31.413508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.414124 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.414128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:31.414245 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:31.414928 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:31.414945 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:31.418109 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":257,"threads_started":1}
W20260619 02:57:31.418563 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:31.410470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:44341 (local address 127.22.230.62:42821)
0619 02:57:31.410697 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:31.410701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:31.410934 (+ 233us) server_negotiation.cc:407] Connection header received
0619 02:57:31.411051 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:31.411054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:31.411108 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:31.411192 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:31.411812 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.412494 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.413611 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.414003 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.414087 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:31.418308 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:44341: BlockingRecv error: recv got EOF from 127.22.230.1:44341 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:57:31 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:31 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:31.707916 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:31.699525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:51349)
0619 02:57:31.700047 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:31.700064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:31.700232 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:31.700592 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:31.700596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:31.701028 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:31.701262 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:31.701271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.702171 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.702174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:31.703114 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:31.703124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.703788 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.703792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:31.703911 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:31.704513 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:31.704529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:31.707691 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":319,"threads_started":1}
W20260619 02:57:31.708073 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:31.699616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:51349 (local address 127.22.230.62:42821)
0619 02:57:31.699804 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:31.699810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:31.700162 (+ 352us) server_negotiation.cc:407] Connection header received
0619 02:57:31.700344 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:31.700349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:31.700441 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:31.700539 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:31.701360 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.702065 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.703224 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:31.703657 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:31.703756 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:31.707865 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:51349: BlockingRecv error: recv got EOF from 127.22.230.3:51349 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:57:32 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:32 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:32.137544 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:32.129079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:55509)
0619 02:57:32.129545 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:32.129560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:32.129712 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:32.130037 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:32.130041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:32.130399 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:32.130548 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:32.130555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.131557 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.131563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:32.132413 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:32.132422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.133083 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.133088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:32.133256 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:32.133771 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:32.133786 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:32.137204 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":233,"threads_started":1}
W20260619 02:57:32.137733 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:32.129141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:55509 (local address 127.22.230.62:42821)
0619 02:57:32.129352 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:32.129359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:32.129635 (+ 276us) server_negotiation.cc:407] Connection header received
0619 02:57:32.129823 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:32.129828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:32.129894 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:32.130008 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:32.130728 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.131428 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.132557 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.132949 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.133074 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:32.137458 (+ 4384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:55509: BlockingRecv error: recv got EOF from 127.22.230.2:55509 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 02:57:32 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:32 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:32.428018 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:32.419370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:37877)
0619 02:57:32.419942 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:32.419958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:32.420070 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:32.420345 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:32.420348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:32.420715 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:32.420866 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:32.420873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.421785 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.421789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:32.422926 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:32.422954 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.423696 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.423700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:32.423835 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:32.424428 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:32.424444 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:32.427786 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":329,"threads_started":1}
W20260619 02:57:32.428194 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:32.419471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:37877 (local address 127.22.230.62:42821)
0619 02:57:32.419622 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:32.419626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:32.420026 (+ 400us) server_negotiation.cc:407] Connection header received
0619 02:57:32.420159 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:32.420162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:32.420217 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:32.420303 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:32.420976 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.421671 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.423167 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.423584 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.423682 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:32.427951 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:37877: BlockingRecv error: recv got EOF from 127.22.230.1:37877 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 02:57:32 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:32 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:32.716967 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:32.708723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:58863)
0619 02:57:32.709250 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:32.709265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:32.709389 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:32.709687 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:32.709691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:32.710007 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:32.710159 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:32.710166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.711108 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.711111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:32.712057 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:32.712071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.712918 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.712922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:32.713079 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:32.713701 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:32.713717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:32.716746 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":259,"threads_started":1}
W20260619 02:57:32.717083 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:32.708822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:58863 (local address 127.22.230.62:42821)
0619 02:57:32.709025 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:32.709032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:32.709328 (+ 296us) server_negotiation.cc:407] Connection header received
0619 02:57:32.709504 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:32.709507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:32.709572 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:32.709658 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:32.710289 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.710988 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.712206 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:32.712782 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:32.712863 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:32.716908 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:58863: BlockingRecv error: recv got EOF from 127.22.230.3:58863 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 02:57:33 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:33 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:33.147330 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:33.138639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:47017)
0619 02:57:33.139177 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:33.139193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:33.139334 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:33.139666 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:33.139670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:33.140039 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:33.140191 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:33.140199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.141211 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.141215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:33.142175 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:33.142186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.142965 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.142971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:33.143160 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:33.143853 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:33.143878 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:33.146982 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
W20260619 02:57:33.147399 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:33.138777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:47017 (local address 127.22.230.62:42821)
0619 02:57:33.138984 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:33.138992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:33.139271 (+ 279us) server_negotiation.cc:407] Connection header received
0619 02:57:33.139459 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:33.139465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:33.139530 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:33.139627 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:33.140324 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.141054 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.142307 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.142822 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.142963 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:33.147165 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:47017: BlockingRecv error: recv got EOF from 127.22.230.2:47017 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 02:57:33 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:33 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:33.437415 32713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:33.428864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:53391)
0619 02:57:33.429371 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:33.429386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:33.429532 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:33.429853 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:33.429858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:33.430262 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:33.430409 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:33.430417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.431396 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.431404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:33.432293 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:33.432303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.432970 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.432975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:33.433149 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:33.433710 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:33.433725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:33.437205 (+ 3480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":255,"threads_started":1}
W20260619 02:57:33.437628 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:33.428934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:53391 (local address 127.22.230.62:42821)
0619 02:57:33.429151 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:33.429160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:33.429471 (+ 311us) server_negotiation.cc:407] Connection header received
0619 02:57:33.429659 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:33.429666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:33.429728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:33.429826 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:33.430546 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.431277 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.432426 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.432811 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.432967 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:33.437382 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:53391: BlockingRecv error: recv got EOF from 127.22.230.1:53391 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 02:57:33 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:33 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:33.726467 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:33.717880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:37141)
0619 02:57:33.718336 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:33.718352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:33.718457 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:33.718826 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:33.718831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:33.719418 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:33.719591 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:33.719600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.720501 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.720505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:33.721499 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:33.721508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.722160 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.722164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:33.722296 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:33.722939 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:33.722962 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:33.726152 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":258,"threads_started":1}
W20260619 02:57:33.726668 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:33.717904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:37141 (local address 127.22.230.62:42821)
0619 02:57:33.718097 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:33.718107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:33.718423 (+ 316us) server_negotiation.cc:407] Connection header received
0619 02:57:33.718534 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:33.718537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:33.718662 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:33.718772 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:33.719706 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.720390 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.721603 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:33.722045 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:33.722132 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:33.726347 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:37141: BlockingRecv error: recv got EOF from 127.22.230.3:37141 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 02:57:34 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:34 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:34.155923 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:34.148252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:57101)
0619 02:57:34.148684 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:34.148699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:34.148816 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:34.149050 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:34.149053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:34.149457 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:34.149595 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:34.149603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.150500 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.150504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:34.151391 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:34.151401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.151976 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.151980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:34.152090 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:34.152580 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:34.152596 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:34.155619 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":221,"threads_started":1}
W20260619 02:57:34.155997 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:34.148435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57101 (local address 127.22.230.62:42821)
0619 02:57:34.148637 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:34.148645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:34.148783 (+ 138us) server_negotiation.cc:407] Connection header received
0619 02:57:34.148903 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:34.148906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:34.148962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:34.149039 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:34.149715 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.150401 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.151497 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.151868 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.151956 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:34.155796 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:57101: BlockingRecv error: recv got EOF from 127.22.230.2:57101 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 02:57:34 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:34 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:34.450284 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:34.438273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:32869)
0619 02:57:34.438774 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:34.438789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:34.438894 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:34.439180 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:34.439184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:34.439562 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:34.439744 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:34.439751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.440710 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.440732 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:34.441591 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:34.441600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.442230 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.442234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:34.442346 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:34.446470 (+ 4124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:34.446487 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:34.450052 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":247,"threads_started":1}
W20260619 02:57:34.450465 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:34.438327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:32869 (local address 127.22.230.62:42821)
0619 02:57:34.438488 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:34.438493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:34.438858 (+ 365us) server_negotiation.cc:407] Connection header received
0619 02:57:34.439000 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:34.439003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:34.439068 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:34.439144 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:34.439869 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.440606 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.441712 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.442116 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.442200 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:34.450227 (+ 8027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:32869: BlockingRecv error: recv got EOF from 127.22.230.1:32869 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 02:57:34 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:34 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:34.736397 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:34.727422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:52941)
0619 02:57:34.727898 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:34.727914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:34.728064 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:34.728407 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:34.728412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:34.728827 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:34.729059 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:34.729069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.730135 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.730142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:34.731108 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:34.731118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.731780 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.731785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:34.731962 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:34.732553 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:34.732571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:34.736175 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
W20260619 02:57:34.736649 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:34.727506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:52941 (local address 127.22.230.62:42821)
0619 02:57:34.727730 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:34.727738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:34.727993 (+ 255us) server_negotiation.cc:407] Connection header received
0619 02:57:34.728187 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:34.728192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:34.728275 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:34.728375 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:34.729197 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.730014 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.731242 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:34.731646 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:34.731771 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:34.736380 (+ 4609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:52941: BlockingRecv error: recv got EOF from 127.22.230.3:52941 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 02:57:35 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:35 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:35.164691 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:35.156810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:39229)
0619 02:57:35.157269 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:35.157284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:35.157390 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:35.157637 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:35.157640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:35.158025 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:35.158191 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:35.158199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.159109 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.159112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:35.160015 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:35.160025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.160595 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.160599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:35.160732 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:35.161274 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:35.161290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:35.164401 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":259,"threads_started":1}
W20260619 02:57:35.164794 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:35.156875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:39229 (local address 127.22.230.62:42821)
0619 02:57:35.157030 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:35.157034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:35.157350 (+ 316us) server_negotiation.cc:407] Connection header received
0619 02:57:35.157467 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:35.157470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:35.157524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:35.157599 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:35.158302 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.158998 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.160123 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.160471 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.160552 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:35.164571 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:39229: BlockingRecv error: recv got EOF from 127.22.230.2:39229 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 02:57:35 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:35 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:35.460026 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:35.451234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:34745)
0619 02:57:35.451754 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:35.451769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:35.451914 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:35.452229 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:35.452235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:35.452624 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:35.452766 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:35.452773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.453690 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.453697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:35.454685 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:35.454695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.455366 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.455372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:35.455547 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:35.456243 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:35.456263 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:35.459703 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":233,"threads_started":1}
W20260619 02:57:35.460076 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:35.451285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:34745 (local address 127.22.230.62:42821)
0619 02:57:35.451444 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:35.451448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:35.451850 (+ 402us) server_negotiation.cc:407] Connection header received
0619 02:57:35.452024 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:35.452029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:35.452101 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:35.452195 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:35.452895 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.453549 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.454830 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.455218 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.455341 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:35.459874 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:34745: BlockingRecv error: recv got EOF from 127.22.230.1:34745 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:57:35 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:35 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:35.746297 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:35.737687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:53601)
0619 02:57:35.738142 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:35.738166 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:35.738293 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:35.738615 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:35.738620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:35.739118 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:35.739321 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:35.739331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.740313 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.740319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:35.741449 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:35.741461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.742066 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.742070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:35.742192 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:35.742796 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:35.742812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:35.745996 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":215,"threads_started":1}
W20260619 02:57:35.746469 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:35.737762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:53601 (local address 127.22.230.62:42821)
0619 02:57:35.737969 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:35.737975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:35.738242 (+ 267us) server_negotiation.cc:407] Connection header received
0619 02:57:35.738387 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:35.738390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:35.738463 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:35.738540 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:35.739441 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.740164 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.741559 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:35.741947 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:35.742035 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:35.746205 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:53601: BlockingRecv error: recv got EOF from 127.22.230.3:53601 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:57:36 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:36 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:36.173780 32721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:36.165729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:36539)
0619 02:57:36.166261 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:36.166277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:36.166387 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:36.166664 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:36.166667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:36.167081 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:36.167250 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:36.167258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.168160 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.168163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:36.169083 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:36.169093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.169690 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.169693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:36.169795 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:36.170374 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:36.170391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:36.173488 (+ 3097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":256,"threads_started":1}
W20260619 02:57:36.173908 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:36.165777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:36539 (local address 127.22.230.62:42821)
0619 02:57:36.166017 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:36.166025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:36.166352 (+ 327us) server_negotiation.cc:407] Connection header received
0619 02:57:36.166476 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:36.166479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:36.166537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:36.166630 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:36.167362 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.168052 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.169189 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.169578 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.169669 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:36.173703 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:36539: BlockingRecv error: recv got EOF from 127.22.230.2:36539 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 02:57:36 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:36 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:36.469513 32722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:36.461114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:44129)
0619 02:57:36.461601 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:36.461615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:36.461765 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:36.462213 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:36.462216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:36.462553 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:36.462772 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:36.462780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.463723 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.463730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:36.464633 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:36.464652 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.465355 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.465362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:36.465508 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:36.466116 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:36.466132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:36.469303 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":242,"threads_started":1}
W20260619 02:57:36.469691 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:36.461159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:44129 (local address 127.22.230.62:42821)
0619 02:57:36.461385 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:36.461393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:36.461703 (+ 310us) server_negotiation.cc:407] Connection header received
0619 02:57:36.461908 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:36.461912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:36.461980 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:36.462043 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:36.462896 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.463589 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.464774 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.465187 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.465302 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:36.469452 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:44129: BlockingRecv error: recv got EOF from 127.22.230.1:44129 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 02:57:36 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:36 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:36.755543 32723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:36.747297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:47621)
0619 02:57:36.747832 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:36.747850 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:36.747959 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:36.748221 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:36.748224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:36.748632 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:36.748790 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:36.748798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.749679 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.749682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:36.750624 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:36.750634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.751378 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.751384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:36.751544 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:36.752136 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:36.752152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:36.755216 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":267,"threads_started":1}
W20260619 02:57:36.755586 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:36.747386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:47621 (local address 127.22.230.62:42821)
0619 02:57:36.747586 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:36.747596 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:36.747923 (+ 327us) server_negotiation.cc:407] Connection header received
0619 02:57:36.748036 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:36.748039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:36.748106 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:36.748185 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:36.748899 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.749576 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.750763 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:36.751232 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:36.751362 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:36.755385 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:47621: BlockingRecv error: recv got EOF from 127.22.230.3:47621 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:57:37 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:37 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:37.184448 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:37.175003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:57149)
0619 02:57:37.175517 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:37.175533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:37.175667 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:37.175968 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:37.175973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:37.176366 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:37.176514 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:37.176522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.177690 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.177694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:37.178734 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:37.178745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.179832 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.179840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:37.180006 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:37.180608 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:37.180629 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:37.184122 (+ 3493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":286,"threads_started":1}
W20260619 02:57:37.184514 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:37.175052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57149 (local address 127.22.230.62:42821)
0619 02:57:37.175207 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:37.175212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:37.175603 (+ 391us) server_negotiation.cc:407] Connection header received
0619 02:57:37.175757 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:37.175760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:37.175818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:37.175896 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:37.176625 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.177576 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.178851 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.179445 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.179621 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:37.184298 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:57149: BlockingRecv error: recv got EOF from 127.22.230.2:57149 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 02:57:37 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:37 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:37.478847 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:37.470481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:36985)
0619 02:57:37.471014 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:37.471029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:37.471131 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:37.471444 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:37.471447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:37.471842 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:37.471989 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:37.471997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.472883 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.472886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:37.473792 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:37.473802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.474402 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.474406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:37.474513 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:37.475111 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:37.475127 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:37.478446 (+ 3319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":325,"threads_started":1}
W20260619 02:57:37.478922 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:37.470530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:36985 (local address 127.22.230.62:42821)
0619 02:57:37.470786 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:37.470795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:37.471103 (+ 308us) server_negotiation.cc:407] Connection header received
0619 02:57:37.471219 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:37.471223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:37.471326 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:37.471409 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:37.472103 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.472798 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.473897 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.474292 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.474375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:37.478689 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:36985: BlockingRecv error: recv got EOF from 127.22.230.1:36985 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 02:57:37 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:37 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:37.764979 32726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:37.756439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:37331)
0619 02:57:37.756987 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:37.757007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:37.757150 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:37.757472 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:37.757476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:37.757854 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:37.758004 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:37.758011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.759008 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.759011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:37.759910 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:37.759920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.760669 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.760676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:37.760822 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:37.761532 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:37.761550 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:37.764750 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":287,"threads_started":1}
W20260619 02:57:37.765210 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:37.756525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:37331 (local address 127.22.230.62:42821)
0619 02:57:37.756682 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:37.756686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:37.757099 (+ 413us) server_negotiation.cc:407] Connection header received
0619 02:57:37.757273 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:37.757276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:37.757356 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:37.757434 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:37.758128 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.758897 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.760038 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:37.760491 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:37.760600 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:37.764956 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:37331: BlockingRecv error: recv got EOF from 127.22.230.3:37331 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 02:57:38 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:38 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:38.194253 32727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:38.185593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:53315)
0619 02:57:38.186070 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:38.186092 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:38.186194 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:38.186458 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:38.186461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:38.187045 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:38.187238 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:38.187249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.188407 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.188411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:38.189333 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:38.189343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.189972 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.189977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:38.190093 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:38.190665 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:38.190686 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:38.194003 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":268,"threads_started":1}
W20260619 02:57:38.194347 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:38.185674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:53315 (local address 127.22.230.62:42821)
0619 02:57:38.185863 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:38.185868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:38.186162 (+ 294us) server_negotiation.cc:407] Connection header received
0619 02:57:38.186284 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:38.186287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:38.186343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:38.186422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:38.187394 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.188297 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.189454 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.189856 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.189945 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:38.194159 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:53315: BlockingRecv error: recv got EOF from 127.22.230.2:53315 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 02:57:38 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:38 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:38.487949 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:38.479731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:46667)
0619 02:57:38.480148 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:38.480163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:38.480264 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:38.480530 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:38.480533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:38.480884 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:38.481017 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:38.481024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.482026 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.482029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:38.483026 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:38.483037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.483680 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.483685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:38.483910 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:38.484570 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:38.484588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:38.487705 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":229,"threads_started":1}
W20260619 02:57:38.488124 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:38.479758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:46667 (local address 127.22.230.62:42821)
0619 02:57:38.479957 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:38.479967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:38.480234 (+ 267us) server_negotiation.cc:407] Connection header received
0619 02:57:38.480351 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:38.480355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:38.480411 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:38.480485 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:38.481148 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.481904 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.483136 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.483506 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.483601 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:38.487866 (+ 4265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:46667: BlockingRecv error: recv got EOF from 127.22.230.1:46667 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 02:57:38 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:38 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:38.774024 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:38.765833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:37219)
0619 02:57:38.766250 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:38.766265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:38.766364 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:38.766622 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:38.766625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:38.767052 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:38.767247 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:38.767258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.768428 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.768436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:38.769381 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:38.769393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.770033 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.770037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:38.770150 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:38.770744 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:38.770761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:38.773804 (+ 3043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":233,"threads_started":1}
W20260619 02:57:38.774112 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:38.766019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:37219 (local address 127.22.230.62:42821)
0619 02:57:38.766233 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:38.766240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:38.766332 (+ 92us) server_negotiation.cc:407] Connection header received
0619 02:57:38.766451 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:38.766454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:38.766508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:38.766604 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:38.767381 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.768271 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.769495 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:38.769905 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:38.769992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:38.773953 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:37219: BlockingRecv error: recv got EOF from 127.22.230.3:37219 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 02:57:39 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:39 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:39.203467 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:39.195083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:35565)
0619 02:57:39.195738 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:39.195759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:39.195889 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:39.196210 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:39.196214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:39.196589 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:39.196753 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:39.196762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.197678 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.197686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:39.198724 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:39.198735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.199415 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.199421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:39.199581 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:39.200166 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:39.200183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:39.203255 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":409,"threads_started":1}
W20260619 02:57:39.203660 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:39.195129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:35565 (local address 127.22.230.62:42821)
0619 02:57:39.195364 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:39.195373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:39.195828 (+ 455us) server_negotiation.cc:407] Connection header received
0619 02:57:39.196014 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:39.196024 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:39.196080 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:39.196174 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:39.196888 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.197544 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.198848 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.199279 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.199403 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:39.203403 (+ 4000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:35565: BlockingRecv error: recv got EOF from 127.22.230.2:35565 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:57:39 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:39 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:39.501101 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:39.488741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:35695)
0619 02:57:39.489183 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:39.489198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:39.489304 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:39.489555 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:39.489558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:39.489915 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:39.490096 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:39.490104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.491012 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.491016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:39.491948 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:39.491958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.492557 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.492561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:39.492683 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:39.497523 (+ 4840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:39.497540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:39.500759 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":196,"threads_started":1}
W20260619 02:57:39.501181 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:39.488924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:35695 (local address 127.22.230.62:42821)
0619 02:57:39.489159 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:39.489165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:39.489255 (+ 90us) server_negotiation.cc:407] Connection header received
0619 02:57:39.489381 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:39.489384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:39.489438 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:39.489512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:39.490224 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.490910 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.492055 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.492449 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.492530 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:39.500938 (+ 8408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:35695: BlockingRecv error: recv got EOF from 127.22.230.1:35695 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 19 02:57:39 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:39 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:39.783205 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:39.774864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:49293)
0619 02:57:39.775414 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:39.775433 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:39.775565 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:39.775917 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:39.775923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:39.776422 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:39.776599 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:39.776608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.777597 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.777601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:39.778614 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:39.778624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.779218 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.779222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:39.779339 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:39.779934 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:39.779951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:39.782963 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":317,"threads_started":1}
W20260619 02:57:39.783332 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:39.774946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:49293 (local address 127.22.230.62:42821)
0619 02:57:39.775134 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:39.775140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:39.775512 (+ 372us) server_negotiation.cc:407] Connection header received
0619 02:57:39.775696 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:39.775701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:39.775780 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:39.775869 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:39.776716 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.777478 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.778719 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:39.779103 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:39.779191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:39.783122 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:49293: BlockingRecv error: recv got EOF from 127.22.230.3:49293 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 02:57:40 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:40 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:40.213915 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:40.204350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:46637)
0619 02:57:40.204857 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:40.204875 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:40.205029 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:40.205402 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:40.205407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:40.205861 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:40.206052 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:40.206061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.207188 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.207197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:40.208488 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:40.208500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.209333 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.209340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:40.209531 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:40.210173 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:40.210191 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:40.213607 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":279,"threads_started":1}
W20260619 02:57:40.214067 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:40.204459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:46637 (local address 127.22.230.62:42821)
0619 02:57:40.204714 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:40.204722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:40.204981 (+ 259us) server_negotiation.cc:407] Connection header received
0619 02:57:40.205175 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:40.205181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:40.205248 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:40.205352 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:40.206202 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.207058 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.208634 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.209103 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.209289 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:40.213836 (+ 4547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:46637: BlockingRecv error: recv got EOF from 127.22.230.2:46637 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 02:57:40 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:40 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:40.510807 32734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:40.501983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:47121)
0619 02:57:40.502501 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:40.502535 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:40.502702 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:40.503062 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:40.503067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:40.503565 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:40.503753 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:40.503761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.504872 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.504879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:40.505810 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:40.505821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.506505 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.506510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:40.506703 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:40.507298 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:40.507315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:40.510502 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":257,"threads_started":1}
W20260619 02:57:40.510937 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:40.502007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:47121 (local address 127.22.230.62:42821)
0619 02:57:40.502205 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:40.502214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:40.502673 (+ 459us) server_negotiation.cc:407] Connection header received
0619 02:57:40.502813 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:40.502819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:40.502893 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:40.503041 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:40.503891 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.504748 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.505941 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.506366 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.506491 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:40.510729 (+ 4238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:47121: BlockingRecv error: recv got EOF from 127.22.230.1:47121 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 02:57:40 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:40 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:40.792253 32735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:40.784055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:39401)
0619 02:57:40.784652 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:40.784668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:40.784779 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:40.785036 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:40.785038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:40.785417 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:40.785584 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:40.785591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.786495 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.786499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:40.787418 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:40.787428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.788065 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.788069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:40.788189 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:40.788744 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:40.788760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:40.792027 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":385,"threads_started":1}
W20260619 02:57:40.792459 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:40.784092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:39401 (local address 127.22.230.62:42821)
0619 02:57:40.784235 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:40.784240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:40.784741 (+ 501us) server_negotiation.cc:407] Connection header received
0619 02:57:40.784858 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:40.784861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:40.784917 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:40.784993 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:40.785697 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.786393 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.787531 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:40.787933 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:40.788023 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:40.792225 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:39401: BlockingRecv error: recv got EOF from 127.22.230.3:39401 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 02:57:41 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:41 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:41.223278 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:41.214913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:57661)
0619 02:57:41.215442 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:41.215458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:41.215561 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:41.215823 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:41.215826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:41.216187 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:41.216362 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:41.216369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.217240 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.217243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:41.218112 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:41.218122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.218894 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.218898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:41.219032 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:41.219640 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:41.219656 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:41.223064 (+ 3408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":284,"threads_started":1}
W20260619 02:57:41.223474 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:41.214944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57661 (local address 127.22.230.62:42821)
0619 02:57:41.215088 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:41.215092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:41.215528 (+ 436us) server_negotiation.cc:407] Connection header received
0619 02:57:41.215644 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:41.215648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:41.215703 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:41.215778 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:41.216462 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.217115 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.218286 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.218735 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.218849 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:41.223216 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:57661: BlockingRecv error: recv got EOF from 127.22.230.2:57661 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 02:57:41 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:41 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:41.519642 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:41.511717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:49697)
0619 02:57:41.512144 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:41.512159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:41.512263 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:41.512509 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:41.512512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:41.512871 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:41.513016 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:41.513024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.513887 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.513890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:41.514788 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:41.514799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.515419 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.515423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:41.515578 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:41.516098 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:41.516114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:41.519315 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":223,"threads_started":1}
W20260619 02:57:41.519748 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:41.511756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:49697 (local address 127.22.230.62:42821)
0619 02:57:41.511952 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:41.511961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:41.512224 (+ 263us) server_negotiation.cc:407] Connection header received
0619 02:57:41.512337 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:41.512340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:41.512394 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:41.512468 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:41.513119 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.513777 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.514907 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.515287 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.515372 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:41.519493 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:49697: BlockingRecv error: recv got EOF from 127.22.230.1:49697 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:57:41 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:41 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:41.801697 32738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:41.793072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:37803)
0619 02:57:41.793610 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:41.793625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:41.793768 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:41.794087 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:41.794092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:41.794519 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:41.794694 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:41.794702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.795656 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.795664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:41.796491 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:41.796500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.797172 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.797178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:41.797339 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:41.797933 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:41.797959 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:41.801403 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":314,"threads_started":1}
W20260619 02:57:41.801870 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:41.793177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:37803 (local address 127.22.230.62:42821)
0619 02:57:41.793386 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:41.793395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:41.793700 (+ 305us) server_negotiation.cc:407] Connection header received
0619 02:57:41.793886 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:41.793892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:41.793961 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:41.794063 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:41.794823 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.795536 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.796610 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:41.796986 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:41.797092 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:41.801605 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:37803: BlockingRecv error: recv got EOF from 127.22.230.3:37803 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 02:57:42 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:42 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:42.232507 32739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:42.224380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:48899)
0619 02:57:42.224816 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:42.224832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:42.224938 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:42.225221 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:42.225224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:42.225595 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:42.225741 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:42.225749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.226698 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.226702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:42.227719 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:42.227730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.228467 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.228473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:42.228649 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:42.229167 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:42.229183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:42.232301 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
W20260619 02:57:42.232684 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:42.224568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:48899 (local address 127.22.230.62:42821)
0619 02:57:42.224789 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:42.224795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:42.224905 (+ 110us) server_negotiation.cc:407] Connection header received
0619 02:57:42.225029 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:42.225032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:42.225102 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:42.225180 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:42.225852 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.226547 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.227858 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.228331 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.228457 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:42.232466 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:48899: BlockingRecv error: recv got EOF from 127.22.230.2:48899 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 02:57:42 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:42 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:42.529255 32740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:42.520677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:53389)
0619 02:57:42.521178 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:42.521193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:42.521291 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:42.521532 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:42.521546 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:42.521872 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:42.522026 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:42.522033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.523169 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.523173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:42.524200 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:42.524211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.524818 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.524822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:42.524939 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:42.525538 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:42.525561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:42.529029 (+ 3468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
W20260619 02:57:42.529402 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:42.520742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:53389 (local address 127.22.230.62:42821)
0619 02:57:42.520947 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:42.520954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:42.521261 (+ 307us) server_negotiation.cc:407] Connection header received
0619 02:57:42.521367 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:42.521370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:42.521425 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:42.521500 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:42.522135 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.522972 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.524311 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.524696 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.524782 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:42.529181 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:53389: BlockingRecv error: recv got EOF from 127.22.230.1:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 02:57:42 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:42 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.3@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:42.811070 32741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:42.802629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.3:47847)
0619 02:57:42.803095 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:42.803111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:42.803221 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:42.803505 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:42.803510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:42.804040 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:42.804236 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:42.804248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.805391 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.805395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:42.806229 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:42.806238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.806910 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.806914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:42.807056 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:42.807583 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:42.807599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:42.810861 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":256,"threads_started":1}
W20260619 02:57:42.811231 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:42.802668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:47847 (local address 127.22.230.62:42821)
0619 02:57:42.802882 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:42.802886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:42.803187 (+ 301us) server_negotiation.cc:407] Connection header received
0619 02:57:42.803310 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:42.803313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:42.803368 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:42.803445 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:42.804380 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.805280 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.806339 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:42.806792 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:42.806882 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:42.811027 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.3:47847: BlockingRecv error: recv got EOF from 127.22.230.3:47847 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 19 02:57:43 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:43 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.2@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:43.241240 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:43.233551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.2:54485)
0619 02:57:43.233984 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:43.234001 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:43.234107 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:43.234367 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:43.234370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:43.234742 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:43.234911 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:43.234920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.235765 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.235768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:43.236696 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:43.236718 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.237312 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.237316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:43.237435 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.237964 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:43.237980 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:43.241011 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":230,"threads_started":1}
W20260619 02:57:43.241439 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:43.233647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:54485 (local address 127.22.230.62:42821)
0619 02:57:43.233863 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:43.233869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:43.234069 (+ 200us) server_negotiation.cc:407] Connection header received
0619 02:57:43.234204 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:43.234207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:43.234263 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:43.234332 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:43.235015 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.235649 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.236819 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.237210 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.237289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.241171 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.2:54485: BlockingRecv error: recv got EOF from 127.22.230.2:54485 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 02:57:43 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 02:57:43 dist-test-slave-2gms krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.230.1@KRBTEST.COM for oryx/127.22.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 02:57:43.538537 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:43.530265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42821 (local address 127.22.230.1:40665)
0619 02:57:43.530803 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:43.530819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:43.530927 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:43.531205 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:43.531208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:43.531576 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:43.531716 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:43.531724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.532649 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.532654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:43.533754 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:43.533765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.534611 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.534616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:43.534720 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.535285 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:43.535301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:43.538332 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.230.62:42821: Server oryx/127.22.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":322,"threads_started":1}
W20260619 02:57:43.538798 32700 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:43.530364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:40665 (local address 127.22.230.62:42821)
0619 02:57:43.530522 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:43.530527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:43.530891 (+ 364us) server_negotiation.cc:407] Connection header received
0619 02:57:43.531013 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:43.531017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:43.531085 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:43.531168 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:43.531834 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.532515 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.534039 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.534449 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.534541 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.538493 (+ 3952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.230.1:40665: BlockingRecv error: recv got EOF from 127.22.230.1:40665 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:57:43.555789 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32283
I20260619 02:57:43.566991 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:43.556841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.22.230.1:42933)
0619 02:57:43.557429 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:43.557436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:43.557463 (+ 27us) server_negotiation.cc:407] Connection header received
0619 02:57:43.557512 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:43.557517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:43.557739 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:43.557908 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:43.559216 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.560212 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.561260 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.561773 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.561865 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.564098 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:43.564106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:43.564118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:43.564152 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:43.566430 (+ 2278us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.230.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 02:57:43.566733 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45786: Request ticket server kudu/127.22.230.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":308,"threads_started":1}
W20260619 02:57:43.567093 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32283: Not authorized: Client connection negotiation failed: client connection to 127.22.230.1:42933: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.230.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 02:57:43.567198 32745 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45786: Request ticket server kudu/127.22.230.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 02:57:43.567328 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 32283
I20260619 02:57:43.582417 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32434
W20260619 02:57:43.594556 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32434: Not authorized: Client connection negotiation failed: client connection to 127.22.230.2:35441: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.230.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 02:57:43.594561 32746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:43.583644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.22.230.2:35441)
0619 02:57:43.584268 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:43.584277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:43.584311 (+ 34us) server_negotiation.cc:407] Connection header received
0619 02:57:43.584377 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:43.584386 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:43.584679 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:43.584887 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:43.586026 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.587372 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.588457 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.588973 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.589064 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.591493 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:43.591501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:43.591526 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:43.591558 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:43.593940 (+ 2382us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.230.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 02:57:43.594246 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37114: Request ticket server kudu/127.22.230.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":289,"threads_started":1}
I20260619 02:57:43.594734 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 32434
I20260619 02:57:43.611544 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32564
I20260619 02:57:43.623698 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:43.612607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.22.230.3:36713)
0619 02:57:43.613227 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:43.613233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:43.613258 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:43.613317 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:43.613323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:43.613545 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:43.613748 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:43.615058 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.616072 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.617228 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:43.617762 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:43.617896 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:43.620439 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:43.620447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:43.620460 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:43.620502 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:43.623019 (+ 2517us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.230.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 02:57:43.623366 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41622: Request ticket server kudu/127.22.230.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":303,"threads_started":1}
W20260619 02:57:43.623689 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32564: Not authorized: Client connection negotiation failed: client connection to 127.22.230.3:36713: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.230.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 02:57:43.623805 32747 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41622: Request ticket server kudu/127.22.230.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 02:57:43.623848 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 32564
I20260619 02:57:43.640868 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32214
I20260619 02:57:43.642033 32276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:43.758373 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 32214
2026-06-19T02:57:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21279 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T02:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:43Z Disabled control of system clock
I20260619 02:57:43.835060 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35895
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:35895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 02:57:44.154268 32752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:44.154534 32752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:44.154672 32752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:44.161667 32752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:44.161768 32752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:44.161813 32752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:44.161861 32752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:44.161904 32752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:44.171413 32752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:35895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:44.172565 32752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:44.174178 32752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:44.185362 32760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:44.185416 32758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:44.185389 32757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:44.186157 32752 server_base.cc:1034] running on GCE node
I20260619 02:57:44.186880 32752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:44.188079 32752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:44.189273 32752 hybrid_clock.cc:648] HybridClock initialized: now 1781837864189230 us; error 35 us; skew 500 ppm
I20260619 02:57:44.189603 32752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:44.191720 32752 webserver.cc:460] Webserver started at http://127.22.230.62:42237/ using document root <none> and password file <none>
I20260619 02:57:44.192363 32752 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:44.192468 32752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:44.192785 32752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:44.195530 32752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "30a6a3d9ae304d609e2c6f5f8acb086c"
format_stamp: "Formatted at 2026-06-19 02:57:44 on dist-test-slave-2gms"
server_key: "fe8d9829ac1514a445ff54fba9f63017"
server_key_iv: "a5f4804fd11ad03807a8b4a149a5b82c"
server_key_version: "encryptionkey@0"
I20260619 02:57:44.196265 32752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "30a6a3d9ae304d609e2c6f5f8acb086c"
format_stamp: "Formatted at 2026-06-19 02:57:44 on dist-test-slave-2gms"
server_key: "fe8d9829ac1514a445ff54fba9f63017"
server_key_iv: "a5f4804fd11ad03807a8b4a149a5b82c"
server_key_version: "encryptionkey@0"
I20260619 02:57:44.201005 32752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 02:57:44.204166 32766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:44.205356 32752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:57:44.205528 32752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "30a6a3d9ae304d609e2c6f5f8acb086c"
format_stamp: "Formatted at 2026-06-19 02:57:44 on dist-test-slave-2gms"
server_key: "fe8d9829ac1514a445ff54fba9f63017"
server_key_iv: "a5f4804fd11ad03807a8b4a149a5b82c"
server_key_version: "encryptionkey@0"
I20260619 02:57:44.205727 32752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:44.218534 32752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:44.219314 32752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:44.219583 32752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:44.236758 32752 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:35895
I20260619 02:57:44.236776 349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:35895 every 8 connection(s)
I20260619 02:57:44.237936 32752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:44.239711 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 32752
I20260619 02:57:44.239904 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:44.240804 23448 external_mini_cluster.cc:1428] Setting key d4a7b203863f3e8e6fd57ed183dc1a3d
I20260619 02:57:44.242511 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:44.253474 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:44.242863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.22.230.62:35895)
0619 02:57:44.243799 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:44.243809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:44.243846 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:57:44.245255 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:44.245274 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:44.245700 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:44.246004 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:44.246707 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:44.247979 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:44.249320 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:44.249936 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:44.250120 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:44.251291 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:44.251298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:44.251310 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:44.251832 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:44.251907 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:44.251926 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:44.252079 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:44.252290 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 02:57:44.252559 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":381,"threads_started":1}
I20260619 02:57:44.254053 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c: Bootstrap starting.
I20260619 02:57:44.257005 350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:44.257934 350 log.cc:824] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:44.260516 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c: No bootstrap required, opened a new log
I20260619 02:57:44.267485 350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35895 } }
I20260619 02:57:44.267771 350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:44.267843 350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a6a3d9ae304d609e2c6f5f8acb086c, State: Initialized, Role: FOLLOWER
I20260619 02:57:44.268353 350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35895 } }
I20260619 02:57:44.268553 350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:44.268646 350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:44.268761 350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:44.270669 350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35895 } }
I20260619 02:57:44.271059 350 leader_election.cc:302] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a6a3d9ae304d609e2c6f5f8acb086c; no voters:
I20260619 02:57:44.271554 350 leader_election.cc:288] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:44.271763 355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:44.272424 355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [term 1 LEADER]: Becoming Leader. State: Replica: 30a6a3d9ae304d609e2c6f5f8acb086c, State: Running, Role: LEADER
I20260619 02:57:44.272985 355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35895 } }
I20260619 02:57:44.273428 350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:44.275816 356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35895 } } }
I20260619 02:57:44.275786 357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a6a3d9ae304d609e2c6f5f8acb086c. Latest consensus state: current_term: 1 leader_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a6a3d9ae304d609e2c6f5f8acb086c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35895 } } }
I20260619 02:57:44.276237 356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:44.276263 357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:44.277542 361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:44.282462 361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:44.290980 361 catalog_manager.cc:1347] Generated new cluster ID: 142506aa9d64422c8a949bca61408568
I20260619 02:57:44.291090 361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:44.305116 361 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:44.306317 361 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:44.319228 361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a6a3d9ae304d609e2c6f5f8acb086c: Generated new TSK 0
I20260619 02:57:44.320166 361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:57:44.331276 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35895
--builtin_ntp_servers=127.22.230.20:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 02:57:44.644820 374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:44.645044 374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:44.645119 374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:44.652024 374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:44.652103 374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:44.652207 374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:44.660610 374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:44.661747 374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:44.663316 374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:44.675480 379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:44.675544 380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:44.675577 382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:44.676590 374 server_base.cc:1034] running on GCE node
I20260619 02:57:44.677227 374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:44.678104 374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:44.679322 374 hybrid_clock.cc:648] HybridClock initialized: now 1781837864679277 us; error 41 us; skew 500 ppm
I20260619 02:57:44.679626 374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:44.684505 374 webserver.cc:460] Webserver started at http://127.22.230.1:33847/ using document root <none> and password file <none>
I20260619 02:57:44.685163 374 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:44.685289 374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:44.685590 374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:44.688287 374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "daf384e3e5be450d9c446dbe737130fe"
format_stamp: "Formatted at 2026-06-19 02:57:44 on dist-test-slave-2gms"
server_key: "ef4d6f22fd8daf7d84e09d4828834fb9"
server_key_iv: "562402c1b8033240bf55bbfe145a704a"
server_key_version: "encryptionkey@0"
I20260619 02:57:44.689029 374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "daf384e3e5be450d9c446dbe737130fe"
format_stamp: "Formatted at 2026-06-19 02:57:44 on dist-test-slave-2gms"
server_key: "ef4d6f22fd8daf7d84e09d4828834fb9"
server_key_iv: "562402c1b8033240bf55bbfe145a704a"
server_key_version: "encryptionkey@0"
I20260619 02:57:44.693655 374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:57:44.696851 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:44.698055 374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:57:44.698226 374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "daf384e3e5be450d9c446dbe737130fe"
format_stamp: "Formatted at 2026-06-19 02:57:44 on dist-test-slave-2gms"
server_key: "ef4d6f22fd8daf7d84e09d4828834fb9"
server_key_iv: "562402c1b8033240bf55bbfe145a704a"
server_key_version: "encryptionkey@0"
I20260619 02:57:44.698422 374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:44.723352 374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:44.724121 374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:44.724411 374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:44.725239 374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:44.727020 374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:44.727129 374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:44.727236 374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:44.727324 374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:44.759783 374 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:33537
I20260619 02:57:44.759866 500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:33537 every 8 connection(s)
I20260619 02:57:44.761041 374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:44.762708 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 374
I20260619 02:57:44.762893 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:44.763535 23448 external_mini_cluster.cc:1428] Setting key c5674508d7a78557aecab76202a96593
I20260619 02:57:44.766119 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35895
--builtin_ntp_servers=127.22.230.20:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:57:44.774407 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:44.763653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:33841 (local address 127.22.230.62:35895)
0619 02:57:44.764110 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:44.764116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:44.765481 (+ 1365us) server_negotiation.cc:407] Connection header received
0619 02:57:44.767331 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:44.767339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:44.767429 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:44.767550 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:44.768521 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:44.769664 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:44.771241 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:44.771965 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:44.772118 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:44.773442 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:44.773450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:44.773458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:44.773503 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:44.773561 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:44.773581 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:44.773709 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:44.773933 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 02:57:44.774042 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
I20260619 02:57:44.775602 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:44.764294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35895 (local address 127.22.230.1:33841)
0619 02:57:44.765320 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:44.765369 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:44.767078 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:44.767734 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:44.767742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:44.767828 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:44.768352 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:44.768367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:44.769821 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:44.769824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:44.771107 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:44.771118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:44.772177 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:44.772181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:44.772367 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:44.773141 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:57:44.773144 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:44.773180 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:57:44.773254 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:44.773730 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:44.773738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:44.773744 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 02:57:44.773882 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 02:57:44.774175 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":339,"threads_started":1}
I20260619 02:57:44.777047 501 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35895
I20260619 02:57:44.777343 501 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:44.778179 501 heartbeater.cc:511] Master 127.22.230.62:35895 requested a full tablet report, sending...
I20260619 02:57:44.780601 315 ts_manager.cc:194] Registered new tserver with Master: daf384e3e5be450d9c446dbe737130fe (127.22.230.1:33537)
I20260619 02:57:44.782815 315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.1:33841
W20260619 02:57:45.090332 505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:45.090646 505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:45.090742 505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:45.097594 505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:45.097679 505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:45.097783 505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:45.106696 505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:45.107825 505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:45.109555 505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:45.122473 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:45.122387 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:45.122427 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:45.123283 505 server_base.cc:1034] running on GCE node
I20260619 02:57:45.123778 505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:45.124570 505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:45.125803 505 hybrid_clock.cc:648] HybridClock initialized: now 1781837865125748 us; error 43 us; skew 500 ppm
I20260619 02:57:45.126072 505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:45.128166 505 webserver.cc:460] Webserver started at http://127.22.230.2:40285/ using document root <none> and password file <none>
I20260619 02:57:45.128774 505 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:45.128856 505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:45.129101 505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:45.131713 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "dc83700a0a1241f29592555f5395e505"
format_stamp: "Formatted at 2026-06-19 02:57:45 on dist-test-slave-2gms"
server_key: "de9d7b4aced2d5fad138dc9a688492e7"
server_key_iv: "82631aecd2843cdf84b4554f6f750dc4"
server_key_version: "encryptionkey@0"
I20260619 02:57:45.132473 505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "dc83700a0a1241f29592555f5395e505"
format_stamp: "Formatted at 2026-06-19 02:57:45 on dist-test-slave-2gms"
server_key: "de9d7b4aced2d5fad138dc9a688492e7"
server_key_iv: "82631aecd2843cdf84b4554f6f750dc4"
server_key_version: "encryptionkey@0"
I20260619 02:57:45.137045 505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:57:45.140085 519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:45.141268 505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:57:45.141453 505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "dc83700a0a1241f29592555f5395e505"
format_stamp: "Formatted at 2026-06-19 02:57:45 on dist-test-slave-2gms"
server_key: "de9d7b4aced2d5fad138dc9a688492e7"
server_key_iv: "82631aecd2843cdf84b4554f6f750dc4"
server_key_version: "encryptionkey@0"
I20260619 02:57:45.141664 505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:45.171180 505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:45.171970 505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:45.172248 505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:45.173240 505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:45.174705 505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:45.174789 505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:45.174935 505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:45.175026 505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:45.206202 505 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:40817
I20260619 02:57:45.206293 631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:40817 every 8 connection(s)
I20260619 02:57:45.207598 505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:45.213562 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 505
I20260619 02:57:45.213749 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:45.214160 23448 external_mini_cluster.cc:1428] Setting key f4b75160e4f8ffd0fb12f6b042aeb8cd
I20260619 02:57:45.216750 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35895
--builtin_ntp_servers=127.22.230.20:38973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 02:57:45.220223 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.210005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:47297 (local address 127.22.230.62:35895)
0619 02:57:45.210188 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:45.210193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:45.211566 (+ 1373us) server_negotiation.cc:407] Connection header received
0619 02:57:45.213291 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:45.213296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:45.213383 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:45.213487 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:45.214680 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.215551 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.217210 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.217956 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.218108 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.219343 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:45.219351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:45.219358 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:45.219382 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:45.219423 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:45.219441 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:45.219532 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:45.219736 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:57:45.219835 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 02:57:45.221001 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.210539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35895 (local address 127.22.230.2:47297)
0619 02:57:45.211415 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:45.211454 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:45.213016 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:45.213762 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:45.213776 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:45.213897 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:45.214484 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:45.214501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.215736 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.215745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:45.217075 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:45.217086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.218151 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.218156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:45.218335 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.219056 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:57:45.219058 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:45.219093 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:57:45.219162 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:45.219534 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:45.219541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:45.219546 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:57:45.219678 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 02:57:45.219932 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":252,"threads_started":1}
I20260619 02:57:45.222272 632 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35895
I20260619 02:57:45.222590 632 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:45.223431 632 heartbeater.cc:511] Master 127.22.230.62:35895 requested a full tablet report, sending...
I20260619 02:57:45.224848 315 ts_manager.cc:194] Registered new tserver with Master: dc83700a0a1241f29592555f5395e505 (127.22.230.2:40817)
I20260619 02:57:45.225944 315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.2:47297
W20260619 02:57:45.528342 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:45.528609 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:45.528741 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:45.535465 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:45.535584 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:45.535748 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:45.544421 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:38973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:45.545622 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:45.547389 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:45.560225 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:45.560395 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:45.560371 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:45.561262 635 server_base.cc:1034] running on GCE node
I20260619 02:57:45.561802 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:45.562708 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:45.563920 635 hybrid_clock.cc:648] HybridClock initialized: now 1781837865563883 us; error 47 us; skew 500 ppm
I20260619 02:57:45.564218 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:45.566344 635 webserver.cc:460] Webserver started at http://127.22.230.3:44839/ using document root <none> and password file <none>
I20260619 02:57:45.567050 635 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:45.567178 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:45.567481 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:45.570075 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "78d415fc2f124414bc204c9be11b29e6"
format_stamp: "Formatted at 2026-06-19 02:57:45 on dist-test-slave-2gms"
server_key: "1afd872b9f8e348b0517d1b97160fabc"
server_key_iv: "454f44408ca7eeebacd5be828e1cd8e4"
server_key_version: "encryptionkey@0"
I20260619 02:57:45.570895 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "78d415fc2f124414bc204c9be11b29e6"
format_stamp: "Formatted at 2026-06-19 02:57:45 on dist-test-slave-2gms"
server_key: "1afd872b9f8e348b0517d1b97160fabc"
server_key_iv: "454f44408ca7eeebacd5be828e1cd8e4"
server_key_version: "encryptionkey@0"
I20260619 02:57:45.575434 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:57:45.578508 649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:45.579721 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:57:45.579876 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "78d415fc2f124414bc204c9be11b29e6"
format_stamp: "Formatted at 2026-06-19 02:57:45 on dist-test-slave-2gms"
server_key: "1afd872b9f8e348b0517d1b97160fabc"
server_key_iv: "454f44408ca7eeebacd5be828e1cd8e4"
server_key_version: "encryptionkey@0"
I20260619 02:57:45.580068 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:45.599038 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:45.599771 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:45.600055 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:45.600907 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:45.602244 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:45.602353 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:45.602456 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:45.602538 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:45.633788 635 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:45007
I20260619 02:57:45.633872 761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:45007 every 8 connection(s)
I20260619 02:57:45.635092 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:45.640729 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 635
I20260619 02:57:45.640920 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:45.641378 23448 external_mini_cluster.cc:1428] Setting key 30d7ad01b5a41ea12f3dfb935b4ad096
I20260619 02:57:45.647010 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.637640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:49103 (local address 127.22.230.62:35895)
0619 02:57:45.637811 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:45.637815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:45.639208 (+ 1393us) server_negotiation.cc:407] Connection header received
0619 02:57:45.640939 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:45.640945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:45.641028 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:45.641147 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:45.642169 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.642956 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.644449 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.644899 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.645030 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.646310 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:45.646314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:45.646317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:45.646334 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:45.646362 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:45.646373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:45.646439 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:45.646667 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 02:57:45.646733 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:57:45.647941 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.638165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35895 (local address 127.22.230.3:49103)
0619 02:57:45.639027 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:45.639103 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:45.640689 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:45.641350 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:45.641357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:45.641438 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:45.642006 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:45.642022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.643088 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.643095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:45.644312 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:45.644322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.645128 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.645134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:45.645349 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.646023 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:57:45.646026 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:45.646064 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:57:45.646140 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:45.646452 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:45.646459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:45.646464 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 02:57:45.646660 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 02:57:45.646936 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":249,"threads_started":1}
I20260619 02:57:45.649183 762 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35895
I20260619 02:57:45.649451 762 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:45.650142 762 heartbeater.cc:511] Master 127.22.230.62:35895 requested a full tablet report, sending...
I20260619 02:57:45.651428 315 ts_manager.cc:194] Registered new tserver with Master: 78d415fc2f124414bc204c9be11b29e6 (127.22.230.3:45007)
I20260619 02:57:45.652143 315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.3:49103
I20260619 02:57:45.655932 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:57:45.661589 503 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:45.660151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.22.230.62:35895)
0619 02:57:45.660340 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:45.660348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:45.660730 (+ 382us) server_negotiation.cc:407] Connection header received
0619 02:57:45.660912 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:45.660917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:45.660987 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:45.661090 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:45.661435 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43052: BlockingRecv error: recv got EOF from 127.0.0.1:43052 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 02:57:45.662732 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 374
I20260619 02:57:45.669466 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.663511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.22.230.1:33537)
0619 02:57:45.663956 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:45.663964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:45.663992 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:57:45.664040 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:45.664045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:45.664228 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:45.664339 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:45.664640 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.665541 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.666812 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.667314 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.667405 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.668272 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:45.668280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:45.668290 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:45.668828 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:45.668867 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:45.668885 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:45.668966 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:45.669090 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 02:57:45.669199 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":236,"threads_started":1}
I20260619 02:57:45.670622 496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:45.749900 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 374
I20260619 02:57:45.766340 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 505
I20260619 02:57:45.773579 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.767302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.22.230.2:40817)
0619 02:57:45.767776 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:45.767784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:45.767809 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:45.767871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:45.767877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:45.768075 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:45.768211 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:45.768533 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.769502 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.770808 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.771324 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.771427 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.772285 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:45.772293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:45.772305 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:45.772827 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:45.772865 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:45.772891 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:45.772967 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:45.773082 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 02:57:45.773246 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260619 02:57:45.774778 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:45.854230 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 505
I20260619 02:57:45.870342 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 635
I20260619 02:57:45.878633 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:45.871351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.22.230.3:45007)
0619 02:57:45.871940 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:45.871947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:45.871975 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:57:45.872030 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:45.872036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:45.872248 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:45.872409 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:45.872852 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.873942 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.875349 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:45.875879 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:45.876007 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:45.877107 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:45.877115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:45.877127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:45.877699 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:45.877741 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:45.877760 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:45.877878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:45.878043 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 02:57:45.878182 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
I20260619 02:57:45.879951 757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:45.960587 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 635
I20260619 02:57:45.982570 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 32752
I20260619 02:57:45.983928 345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:46.093590 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 32752
2026-06-19T02:57:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2330 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T02:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:46Z Disabled control of system clock
I20260619 02:57:46.164471 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:39163
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:39163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:57:46.476873 781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:46.477154 781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:46.477334 781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:46.484354 781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:46.484452 781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:46.484544 781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:46.484643 781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:46.484720 781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:46.493628 781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:39163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:39163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:46.494902 781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:46.496605 781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:46.508078 786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:46.508121 787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:46.508119 789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:46.509137 781 server_base.cc:1034] running on GCE node
I20260619 02:57:46.509851 781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:46.511210 781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:46.512429 781 hybrid_clock.cc:648] HybridClock initialized: now 1781837866512401 us; error 38 us; skew 500 ppm
I20260619 02:57:46.512759 781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:46.514992 781 webserver.cc:460] Webserver started at http://127.22.230.62:40665/ using document root <none> and password file <none>
I20260619 02:57:46.515633 781 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:46.515753 781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:46.516044 781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:46.518798 781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "c0763507a9384303a5192312e235c409"
format_stamp: "Formatted at 2026-06-19 02:57:46 on dist-test-slave-2gms"
server_key: "f0586e17683ccb382889777c12d0aa22"
server_key_iv: "ab3f00d494089d82bcf87dd6e7213199"
server_key_version: "encryptionkey@0"
I20260619 02:57:46.519541 781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "c0763507a9384303a5192312e235c409"
format_stamp: "Formatted at 2026-06-19 02:57:46 on dist-test-slave-2gms"
server_key: "f0586e17683ccb382889777c12d0aa22"
server_key_iv: "ab3f00d494089d82bcf87dd6e7213199"
server_key_version: "encryptionkey@0"
I20260619 02:57:46.524259 781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:57:46.527418 795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:46.528581 781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:46.528745 781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "c0763507a9384303a5192312e235c409"
format_stamp: "Formatted at 2026-06-19 02:57:46 on dist-test-slave-2gms"
server_key: "f0586e17683ccb382889777c12d0aa22"
server_key_iv: "ab3f00d494089d82bcf87dd6e7213199"
server_key_version: "encryptionkey@0"
I20260619 02:57:46.528941 781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:46.534241 781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:46.535003 781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:46.535282 781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:46.552151 781 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:39163
I20260619 02:57:46.552171 846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:39163 every 8 connection(s)
I20260619 02:57:46.553576 781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:46.557147 847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:46.563043 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 781
I20260619 02:57:46.563247 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:46.563855 23448 external_mini_cluster.cc:1428] Setting key da72443d4216e11202a35d5638fa8008
I20260619 02:57:46.567444 847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409: Bootstrap starting.
I20260619 02:57:46.570516 847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:46.571579 847 log.cc:824] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:46.571794 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:46.565755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.22.230.62:39163)
0619 02:57:46.566762 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:46.566773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:46.566813 (+ 40us) server_negotiation.cc:407] Connection header received
0619 02:57:46.568455 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:46.568477 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:46.568861 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:46.569236 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:46.570227 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:46.570233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:46.570244 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:46.570257 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:46.570307 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:46.570322 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:46.570516 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:46.570672 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 02:57:46.570935 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":342,"threads_started":1}
I20260619 02:57:46.574361 847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409: No bootstrap required, opened a new log
I20260619 02:57:46.581169 847 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0763507a9384303a5192312e235c409" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39163 } }
I20260619 02:57:46.581491 847 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:46.581615 847 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0763507a9384303a5192312e235c409, State: Initialized, Role: FOLLOWER
I20260619 02:57:46.582324 847 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0763507a9384303a5192312e235c409" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39163 } }
I20260619 02:57:46.582648 847 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:46.582810 847 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:46.582993 847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:46.585186 847 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0763507a9384303a5192312e235c409" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39163 } }
I20260619 02:57:46.585654 847 leader_election.cc:302] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0763507a9384303a5192312e235c409; no voters:
I20260619 02:57:46.586265 847 leader_election.cc:288] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:46.586401 852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:46.587581 852 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [term 1 LEADER]: Becoming Leader. State: Replica: c0763507a9384303a5192312e235c409, State: Running, Role: LEADER
I20260619 02:57:46.588116 852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0763507a9384303a5192312e235c409" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39163 } }
I20260619 02:57:46.588495 847 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:46.590889 853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0763507a9384303a5192312e235c409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0763507a9384303a5192312e235c409" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39163 } } }
I20260619 02:57:46.590860 854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0763507a9384303a5192312e235c409. Latest consensus state: current_term: 1 leader_uuid: "c0763507a9384303a5192312e235c409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0763507a9384303a5192312e235c409" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39163 } } }
I20260619 02:57:46.591300 853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:46.591328 854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409 [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:46.592833 860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:46.597271 860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:46.604543 860 catalog_manager.cc:1347] Generated new cluster ID: 94cbeb64fa434af98397f1ea143b936d
I20260619 02:57:46.604655 860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:46.632251 860 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:46.632673 860 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:46.639008 860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0763507a9384303a5192312e235c409: Generated new TSK 0
I20260619 02:57:46.639839 860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:57:46.654214 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:39163
--builtin_ntp_servers=127.22.230.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 02:57:46.999193 871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:46.999465 871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:46.999567 871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:47.007189 871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:47.007278 871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:47.007382 871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:47.016047 871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:47.017189 871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:47.018821 871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:47.031333 876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:47.031486 879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:47.031378 877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:47.032305 871 server_base.cc:1034] running on GCE node
I20260619 02:57:47.032923 871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:47.033861 871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:47.035070 871 hybrid_clock.cc:648] HybridClock initialized: now 1781837867035027 us; error 70 us; skew 500 ppm
I20260619 02:57:47.035370 871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:47.037432 871 webserver.cc:460] Webserver started at http://127.22.230.1:39755/ using document root <none> and password file <none>
I20260619 02:57:47.038149 871 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:47.038268 871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:47.038540 871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:47.042817 871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "0db482e6e22046cfab5a6b75716d77bf"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "51dbb81ba701f2b37df13cd1251e7705"
server_key_iv: "2f5b5f53ba7727f67d3086efceb53b73"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.043483 871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "0db482e6e22046cfab5a6b75716d77bf"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "51dbb81ba701f2b37df13cd1251e7705"
server_key_iv: "2f5b5f53ba7727f67d3086efceb53b73"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.048079 871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:57:47.051103 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.052289 871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:57:47.052449 871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "0db482e6e22046cfab5a6b75716d77bf"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "51dbb81ba701f2b37df13cd1251e7705"
server_key_iv: "2f5b5f53ba7727f67d3086efceb53b73"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.052661 871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:47.057934 871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:47.058696 871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:47.058940 871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:47.059763 871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:47.061046 871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:47.061172 871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.061296 871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:47.061378 871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.092404 871 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:45651
I20260619 02:57:47.092437 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:45651 every 8 connection(s)
I20260619 02:57:47.093677 871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:47.101440 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.096023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:41257 (local address 127.22.230.62:39163)
0619 02:57:47.096418 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:47.096423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:47.097575 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 02:57:47.099262 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:47.099268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:47.099340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:47.099456 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:47.100707 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:47.100713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:47.100717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:47.100735 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:47.100782 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:47.100793 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:47.100911 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:47.101103 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 02:57:47.101166 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":210,"threads_started":1}
I20260619 02:57:47.101933 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 871
I20260619 02:57:47.102105 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:47.102752 23448 external_mini_cluster.cc:1428] Setting key 7bf192318d2bd89957db16fb0f345d2f
I20260619 02:57:47.102892 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.096588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:39163 (local address 127.22.230.1:41257)
0619 02:57:47.097417 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:47.097456 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:47.099008 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:47.099679 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:47.099686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:47.099770 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:47.100429 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:57:47.100431 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:47.100466 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:57:47.100544 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:47.100886 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:47.100896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:47.100904 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:57:47.101177 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 02:57:47.101399 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
I20260619 02:57:47.104280 998 heartbeater.cc:347] Connected to a master server at 127.22.230.62:39163
I20260619 02:57:47.104580 998 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:47.105233 998 heartbeater.cc:511] Master 127.22.230.62:39163 requested a full tablet report, sending...
I20260619 02:57:47.105767 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:39163
--builtin_ntp_servers=127.22.230.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:57:47.107410 811 ts_manager.cc:194] Registered new tserver with Master: 0db482e6e22046cfab5a6b75716d77bf (127.22.230.1:45651)
W20260619 02:57:47.422472 1002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:47.422739 1002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:47.422899 1002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:47.430032 1002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:47.430116 1002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:47.430290 1002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:47.439195 1002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:47.440438 1002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:47.442225 1002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:47.458461 1007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:47.458711 1010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:47.458508 1008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:47.459321 1002 server_base.cc:1034] running on GCE node
I20260619 02:57:47.459877 1002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:47.460719 1002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:47.461927 1002 hybrid_clock.cc:648] HybridClock initialized: now 1781837867461871 us; error 50 us; skew 500 ppm
I20260619 02:57:47.462227 1002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:47.464412 1002 webserver.cc:460] Webserver started at http://127.22.230.2:36505/ using document root <none> and password file <none>
I20260619 02:57:47.465045 1002 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:47.465184 1002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:47.465489 1002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:47.468204 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "559002c94cac43f4908ac2ee97ca125d"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "57daa8cda1098006cbc13a2da45ecac6"
server_key_iv: "02142010c29e82aeab69b0948ae630cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.468938 1002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "559002c94cac43f4908ac2ee97ca125d"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "57daa8cda1098006cbc13a2da45ecac6"
server_key_iv: "02142010c29e82aeab69b0948ae630cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.473773 1002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 02:57:47.477090 1016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.478483 1002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:57:47.478665 1002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "559002c94cac43f4908ac2ee97ca125d"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "57daa8cda1098006cbc13a2da45ecac6"
server_key_iv: "02142010c29e82aeab69b0948ae630cc"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.478863 1002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:47.484108 1002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:47.484774 1002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:47.485009 1002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:47.485862 1002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:47.487287 1002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:47.487403 1002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.487525 1002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:47.487608 1002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.525489 1002 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:39719
I20260619 02:57:47.525529 1128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:39719 every 8 connection(s)
I20260619 02:57:47.526796 1002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:47.532895 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 1002
I20260619 02:57:47.533111 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:47.533756 23448 external_mini_cluster.cc:1428] Setting key 7df082e78b23aa2ce1eb10078e74e0ec
I20260619 02:57:47.535856 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.529558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:58117 (local address 127.22.230.62:39163)
0619 02:57:47.529736 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:47.529740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:47.531493 (+ 1753us) server_negotiation.cc:407] Connection header received
0619 02:57:47.533369 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:47.533374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:47.533443 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:47.533544 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:47.535040 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:47.535045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:47.535049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:47.535068 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:47.535108 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:47.535126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:47.535267 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:47.535452 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 02:57:47.535513 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:57:47.536619 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:39163
--builtin_ntp_servers=127.22.230.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 02:57:47.536775 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.530348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:39163 (local address 127.22.230.2:58117)
0619 02:57:47.531319 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:47.531352 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:47.533077 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:47.533778 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:47.533785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:47.533904 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:47.534722 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:57:47.534725 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:47.534774 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:57:47.534872 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:47.535246 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:47.535255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:47.535262 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 02:57:47.535516 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 02:57:47.535741 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":354,"threads_started":1}
I20260619 02:57:47.538277 1129 heartbeater.cc:347] Connected to a master server at 127.22.230.62:39163
I20260619 02:57:47.538554 1129 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:47.539260 1129 heartbeater.cc:511] Master 127.22.230.62:39163 requested a full tablet report, sending...
I20260619 02:57:47.540803 811 ts_manager.cc:194] Registered new tserver with Master: 559002c94cac43f4908ac2ee97ca125d (127.22.230.2:39719)
W20260619 02:57:47.869773 1132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:47.870029 1132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:47.870191 1132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:47.877058 1132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:47.877175 1132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:47.877351 1132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:47.885982 1132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:39163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:47.887198 1132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:47.888962 1132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:47.901547 1137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:47.901479 1138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:47.901522 1140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:47.902256 1132 server_base.cc:1034] running on GCE node
I20260619 02:57:47.902904 1132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:47.903775 1132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:47.904946 1132 hybrid_clock.cc:648] HybridClock initialized: now 1781837867904912 us; error 27 us; skew 500 ppm
I20260619 02:57:47.905261 1132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:57:47.907397 1132 webserver.cc:460] Webserver started at http://127.22.230.3:38261/ using document root <none> and password file <none>
I20260619 02:57:47.908046 1132 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:47.908159 1132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:47.908481 1132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:47.912389 1132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "679f5f67c536470e96ddd187595f6878"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "c15e9ea88515ee29e0c8817498905c17"
server_key_iv: "fc20d08f946504d830f48f86997e1346"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.913192 1132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "679f5f67c536470e96ddd187595f6878"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "c15e9ea88515ee29e0c8817498905c17"
server_key_iv: "fc20d08f946504d830f48f86997e1346"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.917949 1132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 02:57:47.921108 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.922308 1132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:57:47.922469 1132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "679f5f67c536470e96ddd187595f6878"
format_stamp: "Formatted at 2026-06-19 02:57:47 on dist-test-slave-2gms"
server_key: "c15e9ea88515ee29e0c8817498905c17"
server_key_iv: "fc20d08f946504d830f48f86997e1346"
server_key_version: "encryptionkey@0"
I20260619 02:57:47.922726 1132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:47.928030 1132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:47.928767 1132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:47.929033 1132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:47.929863 1132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:47.931172 1132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:47.931291 1132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.931430 1132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:47.931516 1132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:47.963158 1132 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:46619
I20260619 02:57:47.963240 1258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:46619 every 8 connection(s)
I20260619 02:57:47.964413 1132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:47.965372 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 1132
I20260619 02:57:47.965552 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:47.966096 23448 external_mini_cluster.cc:1428] Setting key eb74b482af3fc403cae2ab5eb2ba763d
I20260619 02:57:47.972661 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.967196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:39209 (local address 127.22.230.62:39163)
0619 02:57:47.967397 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:47.967405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:47.968861 (+ 1456us) server_negotiation.cc:407] Connection header received
0619 02:57:47.970617 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:47.970622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:47.970685 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:47.970797 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:47.971966 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:47.971972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:47.971976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:47.971994 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:47.972031 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:47.972042 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:47.972169 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:47.972345 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 02:57:47.972407 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:57:47.973522 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.967766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:39163 (local address 127.22.230.3:39209)
0619 02:57:47.968730 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:47.968763 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:47.970352 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:47.971011 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:47.971018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:47.971101 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:47.971700 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 02:57:47.971702 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:47.971736 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 02:57:47.971815 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:47.972129 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:47.972139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:47.972147 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 02:57:47.972422 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 02:57:47.972660 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":336,"threads_started":1}
I20260619 02:57:47.974673 1259 heartbeater.cc:347] Connected to a master server at 127.22.230.62:39163
I20260619 02:57:47.974929 1259 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:47.975430 1259 heartbeater.cc:511] Master 127.22.230.62:39163 requested a full tablet report, sending...
I20260619 02:57:47.976668 811 ts_manager.cc:194] Registered new tserver with Master: 679f5f67c536470e96ddd187595f6878 (127.22.230.3:46619)
I20260619 02:57:47.981053 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 02:57:47.987061 1000 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 02:57:47.985565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.22.230.62:39163)
0619 02:57:47.985710 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:47.985715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:47.986170 (+ 455us) server_negotiation.cc:407] Connection header received
0619 02:57:47.986358 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:47.986362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:47.986406 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:47.986499 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:47.986838 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39152: BlockingRecv error: recv got EOF from 127.0.0.1:39152 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:57:47.987843 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 871
I20260619 02:57:47.991438 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:47.988755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.22.230.1:45651)
0619 02:57:47.989252 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:47.989258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:47.989281 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:57:47.989356 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:47.989361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:47.989559 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:47.989698 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:47.990481 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:47.990487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:47.990496 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:47.990510 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:47.990546 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:47.990559 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:47.990773 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:47.991112 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 02:57:47.991185 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260619 02:57:47.992483 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:48.074404 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 871
I20260619 02:57:48.091225 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 1002
I20260619 02:57:48.094679 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:48.092159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.22.230.2:39719)
0619 02:57:48.092631 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:48.092638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:48.092663 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:57:48.092728 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:48.092734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:48.092970 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:48.093108 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:48.093968 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:48.093974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:48.093985 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:48.093997 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:48.094051 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:48.094066 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:48.094172 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:48.094246 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:57:48.094349 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260619 02:57:48.095808 1124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:48.174335 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 1002
I20260619 02:57:48.190130 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 1132
I20260619 02:57:48.194125 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:48.191119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.22.230.3:46619)
0619 02:57:48.191700 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:48.191707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:48.191736 (+ 29us) server_negotiation.cc:407] Connection header received
0619 02:57:48.191792 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:48.191798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:48.192034 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:48.192247 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:48.193393 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:48.193399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:48.193410 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 02:57:48.193422 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:48.193468 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 02:57:48.193482 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:48.193656 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:48.193726 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 02:57:48.193823 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260619 02:57:48.195429 1254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:48.274080 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 1132
I20260619 02:57:48.289973 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 781
I20260619 02:57:48.291110 842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:48.391887 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 781
2026-06-19T02:57:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2298 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:57:48 dist-test-slave-2gms krb5kdc[1277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:57:48 dist-test-slave-2gms krb5kdc[1277](info): set up 2 sockets
Jun 19 02:57:48 dist-test-slave-2gms krb5kdc[1277](info): commencing operation
krb5kdc: starting...
W20260619 02:57:49.475564 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:49 dist-test-slave-2gms krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:57:49.697533 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38559
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:50.011049 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:50.011310 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:50.011482 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:50.018478 1290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:57:50.018630 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:50.018740 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:50.018838 1290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:57:50.018923 1290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:57:50.028354 1290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:50.029572 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:50.030834 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:50.041937 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:50.041903 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:50.042369 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:50.042989 1290 server_base.cc:1034] running on GCE node
I20260619 02:57:50.043669 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:50.044900 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:50.046128 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781837870046088 us; error 47 us; skew 500 ppm
I20260619 02:57:50.046444 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:50 dist-test-slave-2gms krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:50.052386 1290 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:57:50.053736 1290 webserver.cc:460] Webserver started at http://127.22.230.62:38975/ using document root <none> and password file <none>
I20260619 02:57:50.054352 1290 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:50.054489 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:50.054827 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:50.057516 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "10680696b5774564a8e67928100d439e"
format_stamp: "Formatted at 2026-06-19 02:57:50 on dist-test-slave-2gms"
server_key: "8e402b8a72637efbe294eba026a48d44"
server_key_iv: "8364b56e2a6d85fadd17e0f6fd1ec032"
server_key_version: "encryptionkey@0"
I20260619 02:57:50.058245 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "10680696b5774564a8e67928100d439e"
format_stamp: "Formatted at 2026-06-19 02:57:50 on dist-test-slave-2gms"
server_key: "8e402b8a72637efbe294eba026a48d44"
server_key_iv: "8364b56e2a6d85fadd17e0f6fd1ec032"
server_key_version: "encryptionkey@0"
I20260619 02:57:50.063526 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 02:57:50.067989 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:50.069589 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260619 02:57:50.069744 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "10680696b5774564a8e67928100d439e"
format_stamp: "Formatted at 2026-06-19 02:57:50 on dist-test-slave-2gms"
server_key: "8e402b8a72637efbe294eba026a48d44"
server_key_iv: "8364b56e2a6d85fadd17e0f6fd1ec032"
server_key_version: "encryptionkey@0"
I20260619 02:57:50.069963 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:50.097563 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:50.101296 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:50.101588 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:50.119077 1290 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:38559
I20260619 02:57:50.119100 1356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:38559 every 8 connection(s)
I20260619 02:57:50.120333 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:57:50.123709 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:50.127851 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 1290
I20260619 02:57:50.128090 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:57:50.128697 23448 external_mini_cluster.cc:1428] Setting key a46a01a0584954d1c8bec18a0c8ea76e
I20260619 02:57:50.135978 1357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e: Bootstrap starting.
I20260619 02:57:50.139430 1357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:50.140650 1357 log.cc:824] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:50.143445 1357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e: No bootstrap required, opened a new log
Jun 19 02:57:50 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:50.152799 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:50.130628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.22.230.62:38559)
0619 02:57:50.131390 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:50.131405 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:50.131453 (+ 48us) server_negotiation.cc:407] Connection header received
0619 02:57:50.134293 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:50.134320 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:50.134808 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:50.135203 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:50.136531 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:50.138501 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:50.139794 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:50.140814 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:50.140984 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:50.144772 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:50.144781 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:50.144798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:50.144847 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:50.148702 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:50.149305 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:50.149312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:50.149320 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:50.149520 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:50.149855 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:50.149861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:50.149865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:50.150501 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:50.150832 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:50.151094 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:57:50.151470 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":292,"threads_started":1}
I20260619 02:57:50.153869 1357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10680696b5774564a8e67928100d439e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38559 } }
I20260619 02:57:50.154215 1357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:50.154311 1357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10680696b5774564a8e67928100d439e, State: Initialized, Role: FOLLOWER
I20260619 02:57:50.154914 1357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10680696b5774564a8e67928100d439e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38559 } }
I20260619 02:57:50.155150 1357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:57:50.155290 1357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:57:50.155475 1357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:50.157541 1357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10680696b5774564a8e67928100d439e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38559 } }
I20260619 02:57:50.157929 1357 leader_election.cc:302] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10680696b5774564a8e67928100d439e; no voters:
I20260619 02:57:50.158447 1357 leader_election.cc:288] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:57:50.158558 1362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:50.159256 1362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [term 1 LEADER]: Becoming Leader. State: Replica: 10680696b5774564a8e67928100d439e, State: Running, Role: LEADER
I20260619 02:57:50.159775 1362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10680696b5774564a8e67928100d439e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38559 } }
I20260619 02:57:50.160492 1357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:57:50.162400 1364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10680696b5774564a8e67928100d439e. Latest consensus state: current_term: 1 leader_uuid: "10680696b5774564a8e67928100d439e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10680696b5774564a8e67928100d439e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38559 } } }
I20260619 02:57:50.162812 1364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:50.162716 1363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10680696b5774564a8e67928100d439e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10680696b5774564a8e67928100d439e" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38559 } } }
I20260619 02:57:50.163062 1363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e [sys.catalog]: This master's current role is: LEADER
I20260619 02:57:50.167167 1370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:57:50.171329 1370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:57:50.179006 1370 catalog_manager.cc:1347] Generated new cluster ID: 4a737b81099b45fa8a2c118b52106abc
I20260619 02:57:50.179104 1370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:57:50.189522 1370 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:57:50.191251 1370 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:57:50.211908 1370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e: Generated new TSK 0
I20260619 02:57:50.212744 1370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:57:50.269649 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38559
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:50.586985 1383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:50.587234 1383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:50.587324 1383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:50.594239 1383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:50.594339 1383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:50.594446 1383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:57:50.603962 1383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:50.605145 1383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:50.606283 1383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:50.618891 1391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:50.619084 1389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:50.619084 1388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:50.619908 1383 server_base.cc:1034] running on GCE node
I20260619 02:57:50.620525 1383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:50.621441 1383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:50.622668 1383 hybrid_clock.cc:648] HybridClock initialized: now 1781837870622628 us; error 64 us; skew 500 ppm
I20260619 02:57:50.622979 1383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:50 dist-test-slave-2gms krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:50.625939 1383 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:57:50.627358 1383 webserver.cc:460] Webserver started at http://127.22.230.1:41621/ using document root <none> and password file <none>
I20260619 02:57:50.628057 1383 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:50.628211 1383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:50.628505 1383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:50.631446 1383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "869246615b2f47c5b60bce3f1fd3fd9e"
format_stamp: "Formatted at 2026-06-19 02:57:50 on dist-test-slave-2gms"
server_key: "cdca15fc40abddc5f8f33eb708ad6e85"
server_key_iv: "c36e0fb65a3459c725f865ac06b3d4e8"
server_key_version: "encryptionkey@0"
I20260619 02:57:50.632175 1383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "869246615b2f47c5b60bce3f1fd3fd9e"
format_stamp: "Formatted at 2026-06-19 02:57:50 on dist-test-slave-2gms"
server_key: "cdca15fc40abddc5f8f33eb708ad6e85"
server_key_iv: "c36e0fb65a3459c725f865ac06b3d4e8"
server_key_version: "encryptionkey@0"
I20260619 02:57:50.636983 1383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 02:57:50.640264 1398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:50.641508 1383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 02:57:50.641680 1383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "869246615b2f47c5b60bce3f1fd3fd9e"
format_stamp: "Formatted at 2026-06-19 02:57:50 on dist-test-slave-2gms"
server_key: "cdca15fc40abddc5f8f33eb708ad6e85"
server_key_iv: "c36e0fb65a3459c725f865ac06b3d4e8"
server_key_version: "encryptionkey@0"
I20260619 02:57:50.641865 1383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:50.656823 1383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:50.660398 1383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:50.660696 1383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:50.661736 1383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:50.663030 1383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:50.663120 1383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:50.663234 1383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:50.663311 1383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:50.694783 1383 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:35485
I20260619 02:57:50.694892 1510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:35485 every 8 connection(s)
I20260619 02:57:50.696048 1383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:57:50.701733 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 1383
I20260619 02:57:50.701959 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:57:50.702622 23448 external_mini_cluster.cc:1428] Setting key e7e03fd66a81f7efd2d9149d228744af
Jun 19 02:57:50 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837870, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:50.715422 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:50.698471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:54223 (local address 127.22.230.62:38559)
0619 02:57:50.698888 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:50.698892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:50.699853 (+ 961us) server_negotiation.cc:407] Connection header received
0619 02:57:50.701525 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:50.701532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:50.701630 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:50.701753 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:50.703309 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:50.704149 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:50.705684 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:50.706072 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:50.706203 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:50.709659 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:50.709664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:50.709667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:50.709695 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:50.712420 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:50.713067 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:50.713076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:50.713078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:50.713206 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:50.713770 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:50.713774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:50.713776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:50.714074 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:50.714185 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:50.715019 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 02:57:50.715182 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":223,"threads_started":1}
I20260619 02:57:50.716960 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:50.699064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38559 (local address 127.22.230.1:54223)
0619 02:57:50.699704 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:50.699740 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:50.701269 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:50.702018 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:50.702030 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:50.702512 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:50.703148 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:50.703164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:50.704327 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:50.704334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:50.705521 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:50.705532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:50.706276 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:50.706285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:50.706509 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:50.707229 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:50.707248 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:50.709451 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:50.712548 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:50.712557 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:50.712572 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:50.712945 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:50.713319 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:50.713323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:50.713328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:50.713616 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:50.714198 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:50.714209 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:50.714664 (+ 455us) client_negotiation.cc:769] Sending connection context
0619 02:57:50.714945 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 02:57:50.715473 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":250,"threads_started":1}
I20260619 02:57:50.718405 1511 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38559
I20260619 02:57:50.718721 1511 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:50.719426 1511 heartbeater.cc:511] Master 127.22.230.62:38559 requested a full tablet report, sending...
I20260619 02:57:50.721292 1322 ts_manager.cc:194] Registered new tserver with Master: 869246615b2f47c5b60bce3f1fd3fd9e (127.22.230.1:35485)
I20260619 02:57:50.722743 1322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:54223
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:57:50.755887 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38559
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:51.068193 1517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:51.068451 1517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:51.068539 1517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:51.075557 1517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:51.075663 1517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:51.075780 1517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:57:51.085003 1517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:51.086153 1517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:51.087359 1517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:51.100013 1523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:51.100244 1522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:51.100183 1525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:51.101199 1517 server_base.cc:1034] running on GCE node
I20260619 02:57:51.101755 1517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:51.102712 1517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:51.103930 1517 hybrid_clock.cc:648] HybridClock initialized: now 1781837871103885 us; error 38 us; skew 500 ppm
I20260619 02:57:51.104240 1517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:51 dist-test-slave-2gms krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:51.107424 1517 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:57:51.108790 1517 webserver.cc:460] Webserver started at http://127.22.230.2:33497/ using document root <none> and password file <none>
I20260619 02:57:51.109470 1517 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:51.109612 1517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:51.109905 1517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:51.112728 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04"
format_stamp: "Formatted at 2026-06-19 02:57:51 on dist-test-slave-2gms"
server_key: "b86638905c96b765bc5f7a244a9b33fe"
server_key_iv: "610b7d3a5bb7f9eb08d0eb0fbe4c9fc9"
server_key_version: "encryptionkey@0"
I20260619 02:57:51.113481 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04"
format_stamp: "Formatted at 2026-06-19 02:57:51 on dist-test-slave-2gms"
server_key: "b86638905c96b765bc5f7a244a9b33fe"
server_key_iv: "610b7d3a5bb7f9eb08d0eb0fbe4c9fc9"
server_key_version: "encryptionkey@0"
I20260619 02:57:51.118275 1517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260619 02:57:51.122334 1532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:51.123873 1517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:57:51.124044 1517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04"
format_stamp: "Formatted at 2026-06-19 02:57:51 on dist-test-slave-2gms"
server_key: "b86638905c96b765bc5f7a244a9b33fe"
server_key_iv: "610b7d3a5bb7f9eb08d0eb0fbe4c9fc9"
server_key_version: "encryptionkey@0"
I20260619 02:57:51.124243 1517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:51.148201 1517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:51.152275 1517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:51.152508 1517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:51.153322 1517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:51.154495 1517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:51.154639 1517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:51.154736 1517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:51.154780 1517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:51.188580 1517 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:46189
I20260619 02:57:51.188653 1644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:46189 every 8 connection(s)
I20260619 02:57:51.189808 1517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:57:51.196625 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 1517
I20260619 02:57:51.196823 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:57:51.197340 23448 external_mini_cluster.cc:1428] Setting key 924c12ba76bc9d4f9675500e60b119d4
Jun 19 02:57:51 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:51.209998 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.192318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:36953 (local address 127.22.230.62:38559)
0619 02:57:51.192487 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.192491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.193589 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 02:57:51.195351 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.195357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.195444 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.195552 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:51.197256 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.198119 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.200066 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.200461 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.200563 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.204493 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:51.204499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:51.204502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:51.204530 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:51.207127 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.207818 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.207828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.207831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.207949 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.208398 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.208407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.208409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.208632 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:51.208781 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.209447 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.209661 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:57:51.210968 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.192763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38559 (local address 127.22.230.2:36953)
0619 02:57:51.193425 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.193469 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.195094 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.195778 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.195786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.196243 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:51.197078 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.197102 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.198264 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.198268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.199917 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.199934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.200688 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.200697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.201038 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.201742 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:51.201776 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:51.204285 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:51.207261 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:51.207270 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:51.207284 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:51.207690 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:51.208088 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:51.208094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:51.208097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:51.208277 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:51.208769 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:51.208776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:51.209106 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.209351 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.209852 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260619 02:57:51.212373 1645 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38559
I20260619 02:57:51.212656 1645 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:51.213318 1645 heartbeater.cc:511] Master 127.22.230.62:38559 requested a full tablet report, sending...
I20260619 02:57:51.214787 1322 ts_manager.cc:194] Registered new tserver with Master: 99f40eefd2ef423f9fad5fbc7f0ffc04 (127.22.230.2:46189)
I20260619 02:57:51.215538 1322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:36953
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:57:51.247871 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38559
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:57:51.559957 1650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:57:51.560213 1650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:57:51.560405 1650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:57:51.567619 1650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:57:51.567739 1650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:57:51.567925 1650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:57:51.577404 1650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:57:51.578612 1650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:57:51.579876 1650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:57:51.592351 1658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:51.592407 1655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:57:51.592432 1656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:57:51.593204 1650 server_base.cc:1034] running on GCE node
I20260619 02:57:51.593780 1650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:57:51.594635 1650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:57:51.595825 1650 hybrid_clock.cc:648] HybridClock initialized: now 1781837871595788 us; error 34 us; skew 500 ppm
I20260619 02:57:51.596115 1650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:57:51 dist-test-slave-2gms krb5kdc[1277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:57:51.599025 1650 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:57:51.600374 1650 webserver.cc:460] Webserver started at http://127.22.230.3:36611/ using document root <none> and password file <none>
I20260619 02:57:51.601002 1650 fs_manager.cc:359] Metadata directory not provided
I20260619 02:57:51.601132 1650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:57:51.601429 1650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:57:51.604212 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "4a515b4116864e9c9a77e807794d1bae"
format_stamp: "Formatted at 2026-06-19 02:57:51 on dist-test-slave-2gms"
server_key: "8c548c2a0a7f849d7c5b29209519af8c"
server_key_iv: "cfb7987aeeea33a226722b62a57e70dd"
server_key_version: "encryptionkey@0"
I20260619 02:57:51.604933 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "4a515b4116864e9c9a77e807794d1bae"
format_stamp: "Formatted at 2026-06-19 02:57:51 on dist-test-slave-2gms"
server_key: "8c548c2a0a7f849d7c5b29209519af8c"
server_key_iv: "cfb7987aeeea33a226722b62a57e70dd"
server_key_version: "encryptionkey@0"
I20260619 02:57:51.609953 1650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 02:57:51.613133 1665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:51.614349 1650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:57:51.614513 1650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "4a515b4116864e9c9a77e807794d1bae"
format_stamp: "Formatted at 2026-06-19 02:57:51 on dist-test-slave-2gms"
server_key: "8c548c2a0a7f849d7c5b29209519af8c"
server_key_iv: "cfb7987aeeea33a226722b62a57e70dd"
server_key_version: "encryptionkey@0"
I20260619 02:57:51.614729 1650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:57:51.628147 1650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:57:51.632211 1650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:57:51.632501 1650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:57:51.633404 1650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:57:51.634728 1650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:57:51.634799 1650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:51.634958 1650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:57:51.635010 1650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:57:51.667964 1650 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:46099
I20260619 02:57:51.668000 1777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:46099 every 8 connection(s)
I20260619 02:57:51.669487 1650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:57:51.678083 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 1650
I20260619 02:57:51.678280 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:57:51.678737 23448 external_mini_cluster.cc:1428] Setting key a67ea6002055aeb75671030abf3385a6
Jun 19 02:57:51 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:57:51.688288 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.671907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:46637 (local address 127.22.230.62:38559)
0619 02:57:51.672070 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.672074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.673358 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 02:57:51.675298 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.675303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.675411 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.675517 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:51.676911 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.677679 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.679479 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.680082 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.680218 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.683562 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:51.683567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:51.683570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:51.683598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:51.685884 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.686483 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.686487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.686489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.686655 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.687013 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.687017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.687019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.687246 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:51.687345 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.687908 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.688062 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:57:51.689178 1780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.672336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38559 (local address 127.22.230.3:46637)
0619 02:57:51.673214 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.673256 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.675037 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.675750 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.675757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.676211 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:57:51.676743 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.676759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.677856 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.677859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.679338 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.679355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.680277 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.680281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.680555 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.681213 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:57:51.681261 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:57:51.683380 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:57:51.686009 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:51.686015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:51.686025 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:51.686385 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:51.686742 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:57:51.686745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:57:51.686747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:57:51.686905 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:57:51.687340 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:57:51.687346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:57:51.687609 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.687816 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.688156 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":268,"threads_started":1}
I20260619 02:57:51.690434 1778 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38559
I20260619 02:57:51.690784 1778 heartbeater.cc:464] Registering TS with master...
I20260619 02:57:51.691529 1778 heartbeater.cc:511] Master 127.22.230.62:38559 requested a full tablet report, sending...
I20260619 02:57:51.692991 1322 ts_manager.cc:194] Registered new tserver with Master: 4a515b4116864e9c9a77e807794d1bae (127.22.230.3:46099)
I20260619 02:57:51.693662 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:57:51.694028 1322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:46637
I20260619 02:57:51.710005 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.698876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.22.230.62:38559)
0619 02:57:51.699104 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.699113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.699469 (+ 356us) server_negotiation.cc:407] Connection header received
0619 02:57:51.699570 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.699573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.699625 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.699704 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:51.700972 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.701655 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.702845 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.703212 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.703300 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.705471 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:51.705475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:51.705479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:51.705505 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:51.707809 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.708373 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.708377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.708379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.708491 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.708781 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.708784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.708786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.709062 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:51.709220 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.709543 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.709688 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 02:57:51.713806 1322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:57:51.720383 1322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:57:51.725421 1511 heartbeater.cc:503] Master 127.22.230.62:38559 was elected leader, sending a full tablet report...
I20260619 02:57:51.743028 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.731791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.22.230.2:46189)
0619 02:57:51.733120 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.733127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.733149 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:57:51.733250 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.733257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.733508 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.733689 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:51.734898 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.736950 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.740338 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.741633 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.741784 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.741996 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.742493 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.742711 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":1069,"threads_started":1}
I20260619 02:57:51.743089 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.731733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:46189 (local address 127.0.0.1:40734)
0619 02:57:51.732567 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.732614 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.732765 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.734009 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.734020 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.734077 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:51.734672 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.734688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.737114 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.737123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.740082 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.740101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.741820 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.741825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.742032 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.742083 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.742214 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.742702 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":250,"thread_start_us":481,"threads_started":1}
I20260619 02:57:51.746966 1580 tablet_service.cc:1467] Processing CreateTablet for tablet f31546ad20e442a895eeae270668a5d3 (DEFAULT_TABLE table=test-table [id=60149228876742f1a3cb852e44884d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:51.748284 1580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31546ad20e442a895eeae270668a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:51.748793 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.734597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.22.230.1:35485)
0619 02:57:51.741377 (+ 6780us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.741389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.741422 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:57:51.741529 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.741538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.741816 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.742121 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:51.742683 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.743848 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.746208 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.747707 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.747868 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.748089 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.748315 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.748460 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6566,"thread_start_us":6359,"threads_started":1}
I20260619 02:57:51.748847 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.734054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:35485 (local address 127.0.0.1:55232)
0619 02:57:51.734619 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.734634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.734743 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.742096 (+ 7353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.742101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.742133 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:51.742511 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.742521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.743988 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.743996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.746022 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.746038 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.747940 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.747948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.748135 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.748168 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.748273 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.748400 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":1,"thread_start_us":891,"threads_started":1}
I20260619 02:57:51.752348 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.733268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:46099 (local address 127.0.0.1:44482)
0619 02:57:51.733927 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.733946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.734047 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.745287 (+ 11240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.745291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.745315 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:51.745700 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.745713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.747703 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.747709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.749776 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.749792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.751564 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.751572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.751763 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.751796 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.751872 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.752001 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":1,"thread_start_us":429,"threads_started":1}
I20260619 02:57:51.753602 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.733596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.22.230.3:46099)
0619 02:57:51.744584 (+ 10988us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.744596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.744755 (+ 159us) server_negotiation.cc:407] Connection header received
0619 02:57:51.744851 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.744859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.745074 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.745293 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:51.745913 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.747524 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.749944 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.751336 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.752772 (+ 1436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.753012 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.753086 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.753229 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10825,"thread_start_us":8937,"threads_started":1}
I20260619 02:57:51.754702 1446 tablet_service.cc:1467] Processing CreateTablet for tablet f31546ad20e442a895eeae270668a5d3 (DEFAULT_TABLE table=test-table [id=60149228876742f1a3cb852e44884d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:51.756139 1446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31546ad20e442a895eeae270668a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:51.757162 1713 tablet_service.cc:1467] Processing CreateTablet for tablet f31546ad20e442a895eeae270668a5d3 (DEFAULT_TABLE table=test-table [id=60149228876742f1a3cb852e44884d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:57:51.758496 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f31546ad20e442a895eeae270668a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:57:51.761106 1796 tablet_bootstrap.cc:492] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Bootstrap starting.
I20260619 02:57:51.764746 1796 tablet_bootstrap.cc:654] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:51.766027 1796 log.cc:824] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:51.769074 1796 tablet_bootstrap.cc:492] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: No bootstrap required, opened a new log
I20260619 02:57:51.769354 1796 ts_tablet_manager.cc:1397] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 02:57:51.770462 1798 tablet_bootstrap.cc:492] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Bootstrap starting.
I20260619 02:57:51.772500 1799 tablet_bootstrap.cc:492] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Bootstrap starting.
I20260619 02:57:51.774075 1798 tablet_bootstrap.cc:654] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:51.775489 1799 tablet_bootstrap.cc:654] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Neither blocks nor log segments found. Creating new log.
I20260619 02:57:51.775533 1798 log.cc:824] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:51.776669 1799 log.cc:824] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Log is configured to *not* fsync() on all Append() calls
I20260619 02:57:51.778419 1798 tablet_bootstrap.cc:492] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: No bootstrap required, opened a new log
I20260619 02:57:51.778709 1798 ts_tablet_manager.cc:1397] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 02:57:51.779083 1799 tablet_bootstrap.cc:492] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: No bootstrap required, opened a new log
I20260619 02:57:51.779326 1799 ts_tablet_manager.cc:1397] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:57:51.779608 1796 raft_consensus.cc:348] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.780025 1796 raft_consensus.cc:374] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:51.780122 1796 raft_consensus.cc:729] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f40eefd2ef423f9fad5fbc7f0ffc04, State: Initialized, Role: FOLLOWER
I20260619 02:57:51.780736 1796 consensus_queue.cc:260] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.781762 1645 heartbeater.cc:503] Master 127.22.230.62:38559 was elected leader, sending a full tablet report...
I20260619 02:57:51.783623 1796 ts_tablet_manager.cc:1428] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260619 02:57:51.787999 1798 raft_consensus.cc:348] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.788369 1798 raft_consensus.cc:374] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:51.788458 1798 raft_consensus.cc:729] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869246615b2f47c5b60bce3f1fd3fd9e, State: Initialized, Role: FOLLOWER
I20260619 02:57:51.788232 1799 raft_consensus.cc:348] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.788691 1799 raft_consensus.cc:374] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:57:51.788789 1799 raft_consensus.cc:729] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a515b4116864e9c9a77e807794d1bae, State: Initialized, Role: FOLLOWER
I20260619 02:57:51.789103 1798 consensus_queue.cc:260] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.789264 1799 consensus_queue.cc:260] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.790686 1798 ts_tablet_manager.cc:1428] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 02:57:51.791280 1802 raft_consensus.cc:482] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:57:51.791361 1778 heartbeater.cc:503] Master 127.22.230.62:38559 was elected leader, sending a full tablet report...
I20260619 02:57:51.791520 1802 raft_consensus.cc:504] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.792018 1799 ts_tablet_manager.cc:1428] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 02:57:51.793228 1802 leader_election.cc:288] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a515b4116864e9c9a77e807794d1bae (127.22.230.3:46099), 869246615b2f47c5b60bce3f1fd3fd9e (127.22.230.1:35485)
I20260619 02:57:51.800419 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.793550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:46099 (local address 127.22.230.2:50321)
0619 02:57:51.794253 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.794278 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.794451 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.794959 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.794964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.794989 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:51.795441 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.795452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.796897 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.796902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.798731 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.798751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.799826 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.799832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.800003 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.800026 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.800091 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.800181 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":383,"threads_started":1}
I20260619 02:57:51.800493 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.793559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:50321 (local address 127.22.230.3:46099)
0619 02:57:51.793896 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.793902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.794352 (+ 450us) server_negotiation.cc:407] Connection header received
0619 02:57:51.794643 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.794648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.794730 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.794810 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:51.795616 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.796781 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.798893 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.799622 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.799737 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.799784 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.800136 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.800266 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260619 02:57:51.801728 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.794823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:45313 (local address 127.22.230.1:35485)
0619 02:57:51.794978 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.794985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.796262 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 02:57:51.796374 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.796379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.796444 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.796538 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:57:51.797032 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.797769 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.799953 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.800970 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.801083 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.801127 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.801436 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.801570 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 02:57:51.801862 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.794376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:35485 (local address 127.22.230.2:45313)
0619 02:57:51.796156 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0619 02:57:51.796175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:57:51.796285 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:57:51.796555 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:57:51.796558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:57:51.796575 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:57:51.796897 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.796906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.797910 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.797915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.799815 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:57:51.799825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.801134 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.801138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:57:51.801297 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.801319 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 02:57:51.801389 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 02:57:51.801464 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":1501,"threads_started":1}
I20260619 02:57:51.802037 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31546ad20e442a895eeae270668a5d3" candidate_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a515b4116864e9c9a77e807794d1bae" is_pre_election: true
I20260619 02:57:51.802606 1733 raft_consensus.cc:2454] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f40eefd2ef423f9fad5fbc7f0ffc04 in term 0.
I20260619 02:57:51.802754 1466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31546ad20e442a895eeae270668a5d3" candidate_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" is_pre_election: true
I20260619 02:57:51.803258 1466 raft_consensus.cc:2454] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f40eefd2ef423f9fad5fbc7f0ffc04 in term 0.
I20260619 02:57:51.803342 1533 leader_election.cc:302] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a515b4116864e9c9a77e807794d1bae, 99f40eefd2ef423f9fad5fbc7f0ffc04; no voters:
I20260619 02:57:51.803723 1802 raft_consensus.cc:2781] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:57:51.803879 1802 raft_consensus.cc:482] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:57:51.803988 1802 raft_consensus.cc:3037] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:51.806036 1802 raft_consensus.cc:504] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.806527 1802 leader_election.cc:288] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [CANDIDATE]: Term 1 election: Requested vote from peers 4a515b4116864e9c9a77e807794d1bae (127.22.230.3:46099), 869246615b2f47c5b60bce3f1fd3fd9e (127.22.230.1:35485)
I20260619 02:57:51.806908 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31546ad20e442a895eeae270668a5d3" candidate_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a515b4116864e9c9a77e807794d1bae"
I20260619 02:57:51.807108 1733 raft_consensus.cc:3037] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:51.807094 1466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f31546ad20e442a895eeae270668a5d3" candidate_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869246615b2f47c5b60bce3f1fd3fd9e"
I20260619 02:57:51.807273 1466 raft_consensus.cc:3037] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:57:51.809180 1733 raft_consensus.cc:2454] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f40eefd2ef423f9fad5fbc7f0ffc04 in term 1.
I20260619 02:57:51.809347 1466 raft_consensus.cc:2454] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f40eefd2ef423f9fad5fbc7f0ffc04 in term 1.
I20260619 02:57:51.809513 1533 leader_election.cc:302] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a515b4116864e9c9a77e807794d1bae, 99f40eefd2ef423f9fad5fbc7f0ffc04; no voters:
I20260619 02:57:51.809782 1802 raft_consensus.cc:2781] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:57:51.810189 1802 raft_consensus.cc:686] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 1 LEADER]: Becoming Leader. State: Replica: 99f40eefd2ef423f9fad5fbc7f0ffc04, State: Running, Role: LEADER
I20260619 02:57:51.810691 1802 consensus_queue.cc:237] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } }
I20260619 02:57:51.814975 1321 catalog_manager.cc:5495] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f40eefd2ef423f9fad5fbc7f0ffc04 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f40eefd2ef423f9fad5fbc7f0ffc04" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 } health_report { overall_health: UNKNOWN } } }
I20260619 02:57:51.844511 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.839803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.22.230.2:46189)
0619 02:57:51.840027 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.840032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.840108 (+ 76us) server_negotiation.cc:407] Connection header received
0619 02:57:51.840276 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.840280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.840350 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.840464 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:57:51.840774 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.841431 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.842711 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.843090 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.843178 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.843556 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:57:51.844011 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:57:51.844119 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.844176 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.844231 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 02:57:51.852243 1466 raft_consensus.cc:1261] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 1 FOLLOWER]: Refusing update from remote peer 99f40eefd2ef423f9fad5fbc7f0ffc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:51.852372 1733 raft_consensus.cc:1261] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 1 FOLLOWER]: Refusing update from remote peer 99f40eefd2ef423f9fad5fbc7f0ffc04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:57:51.853195 1807 consensus_queue.cc:1035] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869246615b2f47c5b60bce3f1fd3fd9e" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:51.853621 1802 consensus_queue.cc:1035] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a515b4116864e9c9a77e807794d1bae" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:57:51.883179 1321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:51.883435 1321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:57:51.886838 1321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10680696b5774564a8e67928100d439e: Sending DeleteTablet for 3 replicas of tablet f31546ad20e442a895eeae270668a5d3
I20260619 02:57:51.887827 1580 tablet_service.cc:1514] Processing DeleteTablet for tablet f31546ad20e442a895eeae270668a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:51 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:40734
I20260619 02:57:51.888155 1713 tablet_service.cc:1514] Processing DeleteTablet for tablet f31546ad20e442a895eeae270668a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:51 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:44482
I20260619 02:57:51.888401 1446 tablet_service.cc:1514] Processing DeleteTablet for tablet f31546ad20e442a895eeae270668a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:57:51 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:55232
I20260619 02:57:51.890446 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 1383
I20260619 02:57:51.918061 1822 tablet_replica.cc:330] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: stopping tablet replica
I20260619 02:57:51.918653 1822 raft_consensus.cc:2229] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:51.919368 1822 raft_consensus.cc:2258] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:51.923797 1821 tablet_replica.cc:330] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: stopping tablet replica
I20260619 02:57:51.924213 1821 raft_consensus.cc:2229] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:57:51.924731 1821 raft_consensus.cc:2258] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:57:51.925572 1820 tablet_replica.cc:330] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: stopping tablet replica
I20260619 02:57:51.925900 1822 ts_tablet_manager.cc:1905] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:51.925992 1820 raft_consensus.cc:2229] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:57:51.926769 1820 raft_consensus.cc:2258] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 02:57:51 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:57:51.930003 1821 ts_tablet_manager.cc:1905] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:51.931529 1820 ts_tablet_manager.cc:1905] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:57:51.933185 1822 ts_tablet_manager.cc:1918] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:51.933350 1822 log.cc:1196] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/f31546ad20e442a895eeae270668a5d3
I20260619 02:57:51.933883 1822 ts_tablet_manager.cc:1939] T f31546ad20e442a895eeae270668a5d3 P 869246615b2f47c5b60bce3f1fd3fd9e: Deleting consensus metadata
I20260619 02:57:51.935726 1306 catalog_manager.cc:4841] TS 869246615b2f47c5b60bce3f1fd3fd9e (127.22.230.1:35485): tablet f31546ad20e442a895eeae270668a5d3 (table test-table [id=60149228876742f1a3cb852e44884d67]) successfully deleted
I20260619 02:57:51.937604 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:51.893044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.22.230.1:35485)
0619 02:57:51.893215 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:51.893223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:51.915111 (+ 21888us) server_negotiation.cc:407] Connection header received
0619 02:57:51.915605 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:51.915612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:51.915701 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:51.915777 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:51.919597 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.920734 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.923370 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:51.924039 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:51.924176 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:51.930148 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:51.930166 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:51.930185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:51.930231 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:51.934074 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.935041 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.935050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.935054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.935208 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:51.935641 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:51.935646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:51.935649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:51.936172 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:51.936465 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:51.937007 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 02:57:51.937238 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:57:51.939126 1820 ts_tablet_manager.cc:1918] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:51.939307 1820 log.cc:1196] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/f31546ad20e442a895eeae270668a5d3
I20260619 02:57:51.939848 1820 ts_tablet_manager.cc:1939] T f31546ad20e442a895eeae270668a5d3 P 99f40eefd2ef423f9fad5fbc7f0ffc04: Deleting consensus metadata
I20260619 02:57:51.940536 1506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:51.941515 1308 catalog_manager.cc:4841] TS 99f40eefd2ef423f9fad5fbc7f0ffc04 (127.22.230.2:46189): tablet f31546ad20e442a895eeae270668a5d3 (table test-table [id=60149228876742f1a3cb852e44884d67]) successfully deleted
I20260619 02:57:51.945617 1821 ts_tablet_manager.cc:1918] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:57:51.945786 1821 log.cc:1196] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/f31546ad20e442a895eeae270668a5d3
I20260619 02:57:51.946215 1821 ts_tablet_manager.cc:1939] T f31546ad20e442a895eeae270668a5d3 P 4a515b4116864e9c9a77e807794d1bae: Deleting consensus metadata
I20260619 02:57:51.947693 1306 catalog_manager.cc:4841] TS 4a515b4116864e9c9a77e807794d1bae (127.22.230.3:46099): tablet f31546ad20e442a895eeae270668a5d3 (table test-table [id=60149228876742f1a3cb852e44884d67]) successfully deleted
I20260619 02:57:52.051609 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 1383
I20260619 02:57:52.071314 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 1517
Jun 19 02:57:52 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:57:52.087661 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:52.072058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.22.230.2:46189)
0619 02:57:52.072227 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:52.072232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:52.072308 (+ 76us) server_negotiation.cc:407] Connection header received
0619 02:57:52.072476 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:52.072480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:52.072541 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:52.072621 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:52.074103 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:52.075005 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:52.076775 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:52.077206 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:52.077297 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:52.082011 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:52.082024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:52.082042 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:52.082078 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:52.085053 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:52.085701 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:52.085708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:52.085711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:52.085862 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:52.086224 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:52.086229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:52.086234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:52.086560 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:52.086933 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:52.087174 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 02:57:52.087325 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 02:57:52.088409 1640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:52.198449 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 1517
I20260619 02:57:52.217311 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 1650
Jun 19 02:57:52 dist-test-slave-2gms krb5kdc[1277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:57:52.231031 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:57:52.218036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.22.230.3:46099)
0619 02:57:52.218206 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:57:52.218210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:57:52.218275 (+ 65us) server_negotiation.cc:407] Connection header received
0619 02:57:52.218411 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:57:52.218414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:57:52.218492 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:57:52.218610 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:57:52.219880 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:52.220592 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:52.221757 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:57:52.222132 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:57:52.222225 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:57:52.225983 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:57:52.225995 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:57:52.226009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:57:52.226038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:57:52.228740 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:52.229292 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:52.229298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:52.229300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:52.229426 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:57:52.229715 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:57:52.229717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:57:52.229719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:57:52.230043 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:57:52.230244 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 02:57:52.230473 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 02:57:52.230710 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:57:52.231613 1773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:52.318725 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 1650
I20260619 02:57:52.335063 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 1290
I20260619 02:57:52.336133 1352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:57:52.440608 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 1290
2026-06-19T02:57:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4049 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:57:52 dist-test-slave-2gms krb5kdc[1830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:57:52 dist-test-slave-2gms krb5kdc[1830](info): set up 2 sockets
Jun 19 02:57:52 dist-test-slave-2gms krb5kdc[1830](info): commencing operation
krb5kdc: starting...
W20260619 02:57:53.527470 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:57:53 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:57:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/rangeradmin_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/rangeradmin_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/rangerlookup_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/rangerlookup_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
I20260619 02:57:53.845083 23448 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrIFYxA/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/postgres -l logfile start
2026-06-19 02:57:56.198 UTC [1860] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:57:56.198 UTC [1860] LOG: listening on IPv4 address "127.22.230.20", port 52861
2026-06-19 02:57:56.202 UTC [1860] LOG: listening on Unix socket "/tmp/.s.PGSQL.52861"
2026-06-19 02:57:56.225 UTC [1863] LOG: database system was shut down at 2026-06-19 02:57:54 UTC
2026-06-19 02:57:56.229 UTC [1860] LOG: database system is ready to accept connections
I20260619 02:57:57.203153 23448 mini_postgres.cc:96] Postgres bound to 52861
2026-06-19 02:57:57.222 UTC [1871] FATAL: database "slave" does not exist
127.22.230.20:52861 - accepting connections
I20260619 02:57:57.223310 23448 mini_ranger.cc:161] Starting Ranger...
I20260619 02:57:57.249737 23448 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:57:57.695611 23448 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:57:57.695742 23448 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-admin
2026-06-19 02:57:57,994 [I] DB FLAVOR :POSTGRES
2026-06-19 02:57:57,994 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:57:57,994 [I] Checking connection..
2026-06-19 02:57:57,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:57:58,250 [I] Checking connection passed.
2026-06-19 02:57:58,250 [I] --------- Verifying version history table ---------
2026-06-19 02:57:58,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:57:58,474 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:57:58,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:57:58,717 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:57:58,717 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:57:58,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:57:58.952 UTC [1935] WARNING: there is no transaction in progress
2026-06-19 02:57:58,958 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:57:58,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:57:59,199 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:57:59,199 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:57:59,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:57:59,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:57:59,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2gms.c.gcp-upstream.internal','N') ;"
2026-06-19 02:57:59,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:58:00,116 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:58:00,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:58:00,340 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:58:00,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:58:00,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:58:00,811 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:58:00,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:58:01.730 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.746 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.762 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.776 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.792 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.809 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.860 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.866 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.879 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.889 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.899 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.912 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.921 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.927 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.934 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:01.940 UTC [2043] WARNING: there is no transaction in progress
2026-06-19 02:58:02,506 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:58:02,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:58:02,752 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:58:02,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:58:02,988 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:58:02,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:58:03,229 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:58:03,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:58:03,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:58:03,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2gms.c.gcp-upstream.internal';"
2026-06-19 02:58:03,931 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:58:03,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:58:04,183 [I] DB_PATCHES have already been applied
I20260619 02:58:04.189323 23448 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrIFYxA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:58:04.189405 23448 mini_ranger.cc:193] Using host: 127.22.230.20
I20260619 02:58:04.205825 23448 mini_ranger.cc:227] Ranger admin URL: http://127.22.230.20:54833
Jun 19, 2026 2:58:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:58:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:58:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:58:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:58:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:58:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:58:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:58:05.332433 23448 mini_ranger.cc:160] Time spent starting Ranger: real 8.109s user 0.000s sys 0.044s
Jun 19, 2026 2:58:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:58:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:58:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:58:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:58:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:58:25.044348 23448 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/rangerkms_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/rangerkms_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.230.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.230.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:58:25 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837905, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:58:25.211808 23448 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:58:25.232656 23448 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:58:26.017459 23448 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:58:26.017633 23448 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms
I20260619 02:58:26.017762 23448 mini_ranger_kms.cc:227] Using postgres at 127.22.230.20:52861
2026-06-19 02:58:26,532 [I] DB FLAVOR :POSTGRES
2026-06-19 02:58:26,533 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:58:26,533 [I] Checking connection
2026-06-19 02:58:26,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:58:26,782 [I] connection success
2026-06-19 02:58:26,783 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:58:26,783 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:58:26,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:58:27,006 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:58:27,007 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:58:27,007 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:58:27,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:52861/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:58:27,279 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:58:27.702241 23448 mini_ranger_kms.cc:310] Created kms service
I20260619 02:58:27.885526 23448 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:58:27.928413 23448 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:58:27 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837905, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:58:28.302500 23448 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:58:28.302857 23448 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:58:28.302925 23448 mini_ranger_kms.cc:240] Using host: 127.22.230.20
I20260619 02:58:28.322292 23448 mini_ranger_kms.cc:276] Ranger KMS PID: 2223
I20260619 02:58:28.322522 23448 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.230.20:50569
Jun 19, 2026 2:58:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:58:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:58:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:58:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:58:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:58:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 02:58:30 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837910, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.230.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:58:30.550041 23448 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.338s user 0.129s sys 0.404s
I20260619 02:58:30.550314 23448 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:58:30.550530 23448 mini_ranger_kms.cc:401] 127.22.230.20:50569/kms/v1/keys
Jun 19 02:58:30 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837910, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.230.20@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
Jun 19 02:58:31 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837910, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.230.20@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
Jun 19 02:58:33 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:58:33.518006 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:43367
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:43367
--ranger_config_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:50569/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:58:33.852934 2274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:58:33.853260 2274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:58:33.853423 2274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:58:33.860631 2274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:58:33.860766 2274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:58:33.860867 2274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:58:33.860972 2274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:58:33.861080 2274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:58:33.871484 2274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.230.20:50569/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:43367
--ranger_config_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:58:33.872751 2274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:58:33.873986 2274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:58:33.887429 2280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:33.887538 2282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:33.887449 2279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:58:33.888666 2274 server_base.cc:1034] running on GCE node
I20260619 02:58:33.889498 2274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:58:33.890954 2274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:58:33.892170 2274 hybrid_clock.cc:648] HybridClock initialized: now 1781837913892125 us; error 42 us; skew 500 ppm
I20260619 02:58:33.892508 2274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:58:33 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:58:33.900758 2274 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:58:33.902376 2274 webserver.cc:460] Webserver started at http://127.22.230.62:46677/ using document root <none> and password file <none>
I20260619 02:58:33.903108 2274 fs_manager.cc:359] Metadata directory not provided
I20260619 02:58:33.903239 2274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:58:33.903592 2274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:58:33 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:58:33.958914 2274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "be4aafc284da46f98efd9358cd002ff9"
format_stamp: "Formatted at 2026-06-19 02:58:33 on dist-test-slave-2gms"
server_key: "4fc408ef02613bf34697fcf362a74dd5"
server_key_iv: "3a86adafc164ef396add21ed4a52f436"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:33.959969 2274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "be4aafc284da46f98efd9358cd002ff9"
format_stamp: "Formatted at 2026-06-19 02:58:33 on dist-test-slave-2gms"
server_key: "4fc408ef02613bf34697fcf362a74dd5"
server_key_iv: "3a86adafc164ef396add21ed4a52f436"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:33.966655 2274 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 02:58:33.982712 2289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:33.984556 2274 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:58:33.984750 2274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "be4aafc284da46f98efd9358cd002ff9"
format_stamp: "Formatted at 2026-06-19 02:58:33 on dist-test-slave-2gms"
server_key: "4fc408ef02613bf34697fcf362a74dd5"
server_key_iv: "3a86adafc164ef396add21ed4a52f436"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:33.984997 2274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:58:34.017072 2274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:58:34.021062 2274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:58:34.021358 2274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:58:34.042852 2274 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:43367
I20260619 02:58:34.042876 2340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:43367 every 8 connection(s)
I20260619 02:58:34.044152 2274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:58:34.046442 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 2274
I20260619 02:58:34.046666 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:58:34.047474 23448 external_mini_cluster.cc:1428] Setting key 65ee22c5284b11d96cbdd6d9488d67ff
I20260619 02:58:34.050047 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 02:58:34 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:58:34.064026 2341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9: Bootstrap starting.
I20260619 02:58:34.068174 2341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9: Neither blocks nor log segments found. Creating new log.
I20260619 02:58:34.069211 2341 log.cc:824] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:58:34.072321 2344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:34.049418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.22.230.62:43367)
0619 02:58:34.050148 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:34.050162 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:34.050209 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:58:34.052384 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:34.052410 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:34.052917 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:34.053325 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:34.054850 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:34.056577 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:34.058362 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:34.059353 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:34.059549 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:34.065176 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:34.065187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:34.065205 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:34.065259 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:34.068509 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:34.069169 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:34.069177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:34.069185 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:34.069368 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:34.069713 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:34.069719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:34.069722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:34.070331 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:34.070651 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:34.071066 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 02:58:34.071432 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":272,"threads_started":1}
I20260619 02:58:34.072554 2341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9: No bootstrap required, opened a new log
I20260619 02:58:34.080247 2341 raft_consensus.cc:348] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4aafc284da46f98efd9358cd002ff9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43367 } }
I20260619 02:58:34.080581 2341 raft_consensus.cc:374] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:58:34.080677 2341 raft_consensus.cc:729] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4aafc284da46f98efd9358cd002ff9, State: Initialized, Role: FOLLOWER
I20260619 02:58:34.081220 2341 consensus_queue.cc:260] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4aafc284da46f98efd9358cd002ff9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43367 } }
I20260619 02:58:34.081460 2341 raft_consensus.cc:388] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:58:34.081595 2341 raft_consensus.cc:482] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:58:34.081797 2341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:58:34.083802 2341 raft_consensus.cc:504] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4aafc284da46f98efd9358cd002ff9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43367 } }
I20260619 02:58:34.084249 2341 leader_election.cc:302] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be4aafc284da46f98efd9358cd002ff9; no voters:
I20260619 02:58:34.084842 2341 leader_election.cc:288] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:58:34.084985 2346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:58:34.086211 2346 raft_consensus.cc:686] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [term 1 LEADER]: Becoming Leader. State: Replica: be4aafc284da46f98efd9358cd002ff9, State: Running, Role: LEADER
I20260619 02:58:34.086824 2346 consensus_queue.cc:237] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4aafc284da46f98efd9358cd002ff9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43367 } }
I20260619 02:58:34.087348 2341 sys_catalog.cc:564] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:58:34.089998 2347 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be4aafc284da46f98efd9358cd002ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4aafc284da46f98efd9358cd002ff9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43367 } } }
I20260619 02:58:34.090024 2348 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be4aafc284da46f98efd9358cd002ff9. Latest consensus state: current_term: 1 leader_uuid: "be4aafc284da46f98efd9358cd002ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4aafc284da46f98efd9358cd002ff9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 43367 } } }
I20260619 02:58:34.090464 2347 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:58:34.090476 2348 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:58:34.091993 2355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:58:34.097910 2355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:58:34.098670 2341 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:58:34.114916 2355 catalog_manager.cc:1347] Generated new cluster ID: 6ef9af541ced47e4bdb530a7479b0887
I20260619 02:58:34.115059 2355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:58:34.142814 2355 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:58:34.144706 2355 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:58:34.161893 2355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9: Generated new TSK 0
I20260619 02:58:34.162951 2355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:58:34.879916 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:50569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43367
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:58:35 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:58:35.309659 2386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:58:35.309949 2386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:58:35.310094 2386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:58:35.318118 2386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:58:35.318251 2386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:58:35.318408 2386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:58:35.331449 2386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.230.20:50569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:58:35.333014 2386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:58:35.334672 2386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:58:35.351241 2396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:35.352393 2397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:35.365010 2399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:58:35.365691 2386 server_base.cc:1034] running on GCE node
I20260619 02:58:35.366549 2386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:58:35.367905 2386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:58:35.369177 2386 hybrid_clock.cc:648] HybridClock initialized: now 1781837915369112 us; error 48 us; skew 500 ppm
I20260619 02:58:35.369622 2386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:58:35 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:58:35.373881 2386 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:58:35.375919 2386 webserver.cc:460] Webserver started at http://127.22.230.1:36117/ using document root <none> and password file <none>
I20260619 02:58:35.376804 2386 fs_manager.cc:359] Metadata directory not provided
I20260619 02:58:35.376929 2386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:58:35.377251 2386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:58:35 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:58:35.401618 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "dbd82d4878dd4478aa54050ed4f49101"
format_stamp: "Formatted at 2026-06-19 02:58:35 on dist-test-slave-2gms"
server_key: "4cde7e0015c82d9786709f8c804415b3"
server_key_iv: "3bc92b020e57c2f356a7fb622848a00e"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:35.402753 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "dbd82d4878dd4478aa54050ed4f49101"
format_stamp: "Formatted at 2026-06-19 02:58:35 on dist-test-slave-2gms"
server_key: "4cde7e0015c82d9786709f8c804415b3"
server_key_iv: "3bc92b020e57c2f356a7fb622848a00e"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:35.413560 2386 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.008s
I20260619 02:58:35.434829 2406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:35.436625 2386 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:58:35.436805 2386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "dbd82d4878dd4478aa54050ed4f49101"
format_stamp: "Formatted at 2026-06-19 02:58:35 on dist-test-slave-2gms"
server_key: "4cde7e0015c82d9786709f8c804415b3"
server_key_iv: "3bc92b020e57c2f356a7fb622848a00e"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:35.437067 2386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:58:35.465127 2386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:58:35.470100 2386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:58:35.470444 2386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:58:35.481808 2386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:58:35.484644 2386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:58:35.484735 2386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:35.484848 2386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:58:35.484912 2386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:35.555292 2386 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:35597
I20260619 02:58:35.555339 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:35597 every 8 connection(s)
I20260619 02:58:35.556825 2386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:58:35.562749 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 2386
I20260619 02:58:35.562935 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:58:35.563738 23448 external_mini_cluster.cc:1428] Setting key 66f4542a3fe207bdac5ab5a6aa6e3f99
Jun 19 02:58:35 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:58:35.601624 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:35.561765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:42989 (local address 127.22.230.62:43367)
0619 02:58:35.575181 (+ 13416us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:35.575186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:35.575204 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:58:35.575254 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:35.575259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:35.575333 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:35.575435 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:35.577501 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:35.578616 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:35.580910 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:35.581544 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:35.581654 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:35.591825 (+ 10171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:35.591836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:35.591844 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:35.591904 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:35.595808 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:35.596983 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:35.596989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:35.596992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:35.597144 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:35.598153 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:35.598158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:35.598161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:35.598492 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:35.598635 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:35.601007 (+ 2372us) server_negotiation.cc:299] Negotiation successful
0619 02:58:35.601207 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13292,"thread_start_us":16812,"threads_started":1}
I20260619 02:58:35.601836 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:35.561163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43367 (local address 127.22.230.1:42989)
0619 02:58:35.571256 (+ 10093us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:35.571315 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:35.573613 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:35.575854 (+ 2241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:35.575865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:35.576461 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:58:35.577312 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:35.577334 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:35.578788 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:35.578793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:35.580737 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:35.580751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:35.581937 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:35.581943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:35.582201 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:35.588622 (+ 6421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:58:35.588653 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:58:35.591581 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:58:35.596186 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:58:35.596195 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:58:35.596212 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:58:35.596836 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:58:35.597279 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:58:35.597284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:58:35.597288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:58:35.598008 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:58:35.599195 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:58:35.599203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:58:35.599623 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 02:58:35.599932 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 02:58:35.600501 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9794,"thread_start_us":12771,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
I20260619 02:58:35.603852 2519 heartbeater.cc:347] Connected to a master server at 127.22.230.62:43367
I20260619 02:58:35.604158 2519 heartbeater.cc:464] Registering TS with master...
I20260619 02:58:35.605000 2519 heartbeater.cc:511] Master 127.22.230.62:43367 requested a full tablet report, sending...
I20260619 02:58:35.608067 2306 ts_manager.cc:194] Registered new tserver with Master: dbd82d4878dd4478aa54050ed4f49101 (127.22.230.1:35597)
I20260619 02:58:35.609973 2306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:42989
Jun 19 02:58:35 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:58:35.650813 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:50569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43367
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:58:35 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837915, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
W20260619 02:58:36.130612 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:58:36.131132 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:58:36.131337 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:58:36.143460 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:58:36.143693 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:58:36.143961 2525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:58:36.159534 2525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.230.20:50569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:58:36.161547 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:58:36.163369 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:58:36.207693 2533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:36.207728 2532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:36.217136 2535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:58:36.223464 2525 server_base.cc:1034] running on GCE node
I20260619 02:58:36.224376 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:58:36.228845 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:58:36.232197 2525 hybrid_clock.cc:648] HybridClock initialized: now 1781837916232081 us; error 114 us; skew 500 ppm
I20260619 02:58:36.232651 2525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:58:36 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:58:36.237166 2525 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:58:36.262892 2525 webserver.cc:460] Webserver started at http://127.22.230.2:42423/ using document root <none> and password file <none>
I20260619 02:58:36.263993 2525 fs_manager.cc:359] Metadata directory not provided
I20260619 02:58:36.264153 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:58:36.264539 2525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:58:36 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:58:36.296216 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "0912ca04c396413ba167f6091d52c5bc"
format_stamp: "Formatted at 2026-06-19 02:58:36 on dist-test-slave-2gms"
server_key: "09e3cc2f72a972c5b24b4395f694cb19"
server_key_iv: "2fa6fa76a8684c01b047adbdcd5856d2"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:36.297487 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "0912ca04c396413ba167f6091d52c5bc"
format_stamp: "Formatted at 2026-06-19 02:58:36 on dist-test-slave-2gms"
server_key: "09e3cc2f72a972c5b24b4395f694cb19"
server_key_iv: "2fa6fa76a8684c01b047adbdcd5856d2"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:36.306849 2525 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260619 02:58:36.331517 2542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:36.336198 2525 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260619 02:58:36.336449 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "0912ca04c396413ba167f6091d52c5bc"
format_stamp: "Formatted at 2026-06-19 02:58:36 on dist-test-slave-2gms"
server_key: "09e3cc2f72a972c5b24b4395f694cb19"
server_key_iv: "2fa6fa76a8684c01b047adbdcd5856d2"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:36.336799 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:58:36.375222 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:58:36.379396 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:58:36.379698 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:58:36.380734 2525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:58:36.382300 2525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:58:36.382376 2525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:36.382500 2525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:58:36.382599 2525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:36.427613 2525 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:40083
I20260619 02:58:36.427634 2661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:40083 every 8 connection(s)
I20260619 02:58:36.428929 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:58:36.435611 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 2525
I20260619 02:58:36.435789 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:58:36.436299 23448 external_mini_cluster.cc:1428] Setting key 23c9e605588358ef986169bfdcbee133
Jun 19 02:58:36 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:58:36.452176 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:36.432341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:36843 (local address 127.22.230.62:43367)
0619 02:58:36.432835 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:36.432839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:36.433744 (+ 905us) server_negotiation.cc:407] Connection header received
0619 02:58:36.435700 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:36.435706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:36.435783 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:36.435900 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:36.437592 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:36.438565 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:36.440759 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:36.441483 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:36.441619 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:36.445786 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:36.445810 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:36.445815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:36.445849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:36.449227 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:36.449872 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:36.449880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:36.449885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:36.450026 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:36.450465 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:36.450471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:36.450474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:36.450812 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:36.450963 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:36.451590 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 02:58:36.451823 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":288,"threads_started":1}
I20260619 02:58:36.453293 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:36.432880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43367 (local address 127.22.230.2:36843)
0619 02:58:36.433603 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:36.433638 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:36.435389 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:36.436164 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:36.436175 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:36.436735 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:58:36.437429 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:36.437447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:36.438764 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:36.438770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:36.440617 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:36.440630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:36.441751 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:36.441759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:36.442064 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:36.442852 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:58:36.442880 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:58:36.445582 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:58:36.449368 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:58:36.449376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:58:36.449397 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:58:36.449757 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:58:36.450136 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:58:36.450142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:58:36.450146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:58:36.450345 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:58:36.450939 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:58:36.450945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:58:36.451252 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 02:58:36.451505 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 02:58:36.451986 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":303,"threads_started":1}
I20260619 02:58:36.454655 2662 heartbeater.cc:347] Connected to a master server at 127.22.230.62:43367
I20260619 02:58:36.454977 2662 heartbeater.cc:464] Registering TS with master...
I20260619 02:58:36.455621 2662 heartbeater.cc:511] Master 127.22.230.62:43367 requested a full tablet report, sending...
I20260619 02:58:36.457031 2306 ts_manager.cc:194] Registered new tserver with Master: 0912ca04c396413ba167f6091d52c5bc (127.22.230.2:40083)
I20260619 02:58:36.457816 2306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:36843
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:58:36.497694 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:50569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43367
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:58:36.613387 2519 heartbeater.cc:503] Master 127.22.230.62:43367 was elected leader, sending a full tablet report...
W20260619 02:58:36.855361 2668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:58:36.855661 2668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:58:36.855824 2668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:58:36.863359 2668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:58:36.863502 2668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:58:36.863682 2668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:58:36.873210 2668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.230.20:50569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:43367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:58:36.874464 2668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:58:36.875748 2668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:58:36.889478 2673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:36.889379 2674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:58:36.889557 2676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:58:36.890529 2668 server_base.cc:1034] running on GCE node
I20260619 02:58:36.891252 2668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:58:36.892204 2668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:58:36.893432 2668 hybrid_clock.cc:648] HybridClock initialized: now 1781837916893390 us; error 56 us; skew 500 ppm
I20260619 02:58:36.893748 2668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:58:36 dist-test-slave-2gms krb5kdc[1830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:58:36.896850 2668 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:58:36.898211 2668 webserver.cc:460] Webserver started at http://127.22.230.3:42951/ using document root <none> and password file <none>
I20260619 02:58:36.898900 2668 fs_manager.cc:359] Metadata directory not provided
I20260619 02:58:36.898999 2668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:58:36.899237 2668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:58:36 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:58:36.915441 2668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "aae42342a8ec42a4aad50074983a5aef"
format_stamp: "Formatted at 2026-06-19 02:58:36 on dist-test-slave-2gms"
server_key: "76744afbe4b99bdf1bf18f91edecf7b4"
server_key_iv: "f0d5c7238eb79ec9aa6cd47fa3c492b5"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:36.916266 2668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "aae42342a8ec42a4aad50074983a5aef"
format_stamp: "Formatted at 2026-06-19 02:58:36 on dist-test-slave-2gms"
server_key: "76744afbe4b99bdf1bf18f91edecf7b4"
server_key_iv: "f0d5c7238eb79ec9aa6cd47fa3c492b5"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:36.922116 2668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 02:58:36.939267 2683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:36.940814 2668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 02:58:36.941015 2668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "aae42342a8ec42a4aad50074983a5aef"
format_stamp: "Formatted at 2026-06-19 02:58:36 on dist-test-slave-2gms"
server_key: "76744afbe4b99bdf1bf18f91edecf7b4"
server_key_iv: "f0d5c7238eb79ec9aa6cd47fa3c492b5"
server_key_version: "kuduclusterkey@0"
I20260619 02:58:36.941242 2668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:58:36.969327 2668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:58:36.973562 2668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:58:36.974274 2668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:58:36.975317 2668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:58:36.976647 2668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:58:36.976717 2668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:36.976836 2668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:58:36.976882 2668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:58:37.018265 2668 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:43107
I20260619 02:58:37.018352 2795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:43107 every 8 connection(s)
I20260619 02:58:37.019500 2668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:58:37.024796 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 2668
I20260619 02:58:37.025000 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:58:37.025684 23448 external_mini_cluster.cc:1428] Setting key 5c5e60d1ce93b1f531dba5bbc7c6dd9e
Jun 19 02:58:37 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837916, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:58:37.040311 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.022176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:45485 (local address 127.22.230.62:43367)
0619 02:58:37.022758 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.022765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.023821 (+ 1056us) server_negotiation.cc:407] Connection header received
0619 02:58:37.025994 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.025999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.026090 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.026204 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:37.028036 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.028931 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.030474 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.030960 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.031100 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.034548 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:37.034554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:37.034557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:37.034632 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:37.037259 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.037948 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.037955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.037958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.038095 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.038493 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.038499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.038505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.038823 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:37.038971 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.039738 (+ 767us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.039954 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":231,"threads_started":1}
I20260619 02:58:37.041436 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.022763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:43367 (local address 127.22.230.3:45485)
0619 02:58:37.023638 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:37.023692 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:37.025714 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:37.026462 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:37.026472 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:37.027092 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:58:37.027867 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.027897 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.029082 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.029088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.030316 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.030327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.031189 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.031196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.031427 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.032097 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:58:37.032117 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:58:37.034327 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:58:37.037404 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:58:37.037412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:58:37.037427 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:58:37.037828 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:58:37.038208 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:58:37.038214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:58:37.038219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:58:37.038386 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:58:37.038946 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:58:37.038953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:58:37.039277 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 02:58:37.039621 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 02:58:37.040098 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":425,"threads_started":1}
I20260619 02:58:37.046690 2796 heartbeater.cc:347] Connected to a master server at 127.22.230.62:43367
I20260619 02:58:37.047010 2796 heartbeater.cc:464] Registering TS with master...
I20260619 02:58:37.047719 2796 heartbeater.cc:511] Master 127.22.230.62:43367 requested a full tablet report, sending...
I20260619 02:58:37.049158 2306 ts_manager.cc:194] Registered new tserver with Master: aae42342a8ec42a4aad50074983a5aef (127.22.230.3:43107)
I20260619 02:58:37.050101 2306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:45485
I20260619 02:58:37.053434 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:58:37.070663 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.058145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.22.230.62:43367)
0619 02:58:37.058305 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.058327 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.058803 (+ 476us) server_negotiation.cc:407] Connection header received
0619 02:58:37.059048 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.059053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.059135 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.059235 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:37.060516 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.061241 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.062629 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.063125 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.063263 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.065595 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:37.065602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:37.065606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:37.065645 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:37.068098 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.068765 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.068772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.068776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.068898 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.069277 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.069283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.069286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.069551 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:37.069795 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.070097 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.070318 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:58:37.074887 2306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:58:37.082334 2306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:58:37.120612 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.095286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:40083 (local address 127.0.0.1:49982)
0619 02:58:37.112044 (+ 16758us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:37.112062 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:37.112190 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:37.113126 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:37.113134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:37.113164 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:58:37.113556 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.113576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.115473 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.115478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.117761 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.117775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.119743 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.119749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.119908 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.119955 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 02:58:37.120078 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 02:58:37.120251 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16641,"thread_start_us":11962,"threads_started":1}
I20260619 02:58:37.121461 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.095497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.22.230.2:40083)
0619 02:58:37.109434 (+ 13937us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.109447 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.112309 (+ 2862us) server_negotiation.cc:407] Connection header received
0619 02:58:37.112418 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.112425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.112656 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.112847 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:58:37.113748 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.115318 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.117932 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.119279 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.119401 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.119596 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.121006 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.121162 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13706,"thread_start_us":13956,"threads_started":1}
I20260619 02:58:37.125782 2597 tablet_service.cc:1467] Processing CreateTablet for tablet 52329b2f52ac41b9b812da479ec4c579 (DEFAULT_TABLE table=test-table [id=4f230c39bc1347c0b7d6a7e4d67c663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:58:37.127701 2597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52329b2f52ac41b9b812da479ec4c579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:58:37.128152 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.095081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:35597 (local address 127.0.0.1:43734)
0619 02:58:37.107653 (+ 12572us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:37.107703 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:37.107911 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:37.118920 (+ 11009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:37.118928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:37.118967 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:58:37.119532 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.119547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.122952 (+ 3405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.122961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.125238 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.125249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.127401 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.127407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.127577 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.127623 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 02:58:37.127686 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 02:58:37.127812 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12303,"thread_start_us":16693,"threads_started":1}
I20260619 02:58:37.130615 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.095272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.22.230.1:35597)
0619 02:58:37.111841 (+ 16569us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.111851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.111884 (+ 33us) server_negotiation.cc:407] Connection header received
0619 02:58:37.112007 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.112018 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.118359 (+ 6341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.119838 (+ 1479us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:58:37.120168 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.122176 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.125638 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.127192 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.129818 (+ 2626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.130055 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.130127 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.130283 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16354,"thread_start_us":16161,"threads_started":1}
I20260619 02:58:37.133805 2454 tablet_service.cc:1467] Processing CreateTablet for tablet 52329b2f52ac41b9b812da479ec4c579 (DEFAULT_TABLE table=test-table [id=4f230c39bc1347c0b7d6a7e4d67c663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:58:37.135165 2454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52329b2f52ac41b9b812da479ec4c579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:58:37.135242 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.112441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43107 (local address 127.0.0.1:33348)
0619 02:58:37.122253 (+ 9812us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:37.122270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:37.122355 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:37.123502 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:37.123506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:37.123534 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:58:37.125407 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.125413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.128340 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.128344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.129581 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.129592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.134435 (+ 4843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.134444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.134644 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.134681 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 02:58:37.134779 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 02:58:37.134915 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9704,"mutex_wait_us":1725,"thread_start_us":11541,"threads_started":1}
I20260619 02:58:37.136222 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.112497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.22.230.3:43107)
0619 02:58:37.122943 (+ 10446us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.122951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.122981 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:58:37.123072 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.123081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.123309 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.123653 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:58:37.125562 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.127562 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.129748 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.131525 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.131652 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.131866 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.135453 (+ 3587us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.135611 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10280,"thread_start_us":10136,"threads_started":1}
I20260619 02:58:37.140519 2731 tablet_service.cc:1467] Processing CreateTablet for tablet 52329b2f52ac41b9b812da479ec4c579 (DEFAULT_TABLE table=test-table [id=4f230c39bc1347c0b7d6a7e4d67c663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:58:37.142370 2731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52329b2f52ac41b9b812da479ec4c579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:58:37.150453 2815 tablet_bootstrap.cc:492] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Bootstrap starting.
I20260619 02:58:37.154505 2815 tablet_bootstrap.cc:654] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Neither blocks nor log segments found. Creating new log.
I20260619 02:58:37.158893 2816 tablet_bootstrap.cc:492] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Bootstrap starting.
I20260619 02:58:37.159305 2815 log.cc:824] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Log is configured to *not* fsync() on all Append() calls
I20260619 02:58:37.165655 2816 tablet_bootstrap.cc:654] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Neither blocks nor log segments found. Creating new log.
I20260619 02:58:37.167374 2816 log.cc:824] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Log is configured to *not* fsync() on all Append() calls
I20260619 02:58:37.168453 2817 tablet_bootstrap.cc:492] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Bootstrap starting.
I20260619 02:58:37.172585 2815 tablet_bootstrap.cc:492] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: No bootstrap required, opened a new log
I20260619 02:58:37.172938 2815 ts_tablet_manager.cc:1397] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.014s
I20260619 02:58:37.173787 2817 tablet_bootstrap.cc:654] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Neither blocks nor log segments found. Creating new log.
I20260619 02:58:37.175593 2817 log.cc:824] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Log is configured to *not* fsync() on all Append() calls
I20260619 02:58:37.180059 2816 tablet_bootstrap.cc:492] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: No bootstrap required, opened a new log
I20260619 02:58:37.180493 2816 ts_tablet_manager.cc:1397] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.011s
I20260619 02:58:37.183912 2815 raft_consensus.cc:348] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.184389 2815 raft_consensus.cc:374] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:58:37.184515 2815 raft_consensus.cc:729] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0912ca04c396413ba167f6091d52c5bc, State: Initialized, Role: FOLLOWER
I20260619 02:58:37.185110 2815 consensus_queue.cc:260] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.185535 2817 tablet_bootstrap.cc:492] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: No bootstrap required, opened a new log
I20260619 02:58:37.186244 2817 ts_tablet_manager.cc:1397] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260619 02:58:37.193686 2816 raft_consensus.cc:348] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.194209 2816 raft_consensus.cc:374] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:58:37.194372 2816 raft_consensus.cc:729] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd82d4878dd4478aa54050ed4f49101, State: Initialized, Role: FOLLOWER
I20260619 02:58:37.195219 2816 consensus_queue.cc:260] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.196229 2662 heartbeater.cc:503] Master 127.22.230.62:43367 was elected leader, sending a full tablet report...
I20260619 02:58:37.199453 2817 raft_consensus.cc:348] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.199994 2817 raft_consensus.cc:374] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:58:37.200142 2817 raft_consensus.cc:729] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae42342a8ec42a4aad50074983a5aef, State: Initialized, Role: FOLLOWER
I20260619 02:58:37.200995 2815 ts_tablet_manager.cc:1428] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Time spent starting tablet: real 0.028s user 0.016s sys 0.005s
I20260619 02:58:37.201052 2816 ts_tablet_manager.cc:1428] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260619 02:58:37.200950 2817 consensus_queue.cc:260] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.204092 2796 heartbeater.cc:503] Master 127.22.230.62:43367 was elected leader, sending a full tablet report...
I20260619 02:58:37.205824 2817 ts_tablet_manager.cc:1428] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Time spent starting tablet: real 0.019s user 0.011s sys 0.006s
I20260619 02:58:37.246604 2821 raft_consensus.cc:482] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:58:37.246922 2821 raft_consensus.cc:504] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.248893 2821 leader_election.cc:288] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbd82d4878dd4478aa54050ed4f49101 (127.22.230.1:35597), aae42342a8ec42a4aad50074983a5aef (127.22.230.3:43107)
I20260619 02:58:37.257570 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.249252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57935 (local address 127.22.230.1:35597)
0619 02:58:37.249582 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.249588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.249914 (+ 326us) server_negotiation.cc:407] Connection header received
0619 02:58:37.250224 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.250233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.250329 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.250403 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:58:37.251336 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.252195 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.255311 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.256514 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.256669 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.256720 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.257192 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.257313 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:58:37.257608 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.249142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:35597 (local address 127.22.230.2:57935)
0619 02:58:37.249819 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:37.249844 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:37.250020 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:37.250677 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:37.250684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:37.250721 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:58:37.251193 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.251203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.252367 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.252374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.255008 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.255029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.256833 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.256838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.257041 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.257064 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 02:58:37.257166 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:58:37.257289 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":361,"threads_started":1}
I20260619 02:58:37.258638 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.250146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:37603 (local address 127.22.230.3:43107)
0619 02:58:37.250325 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.250331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.251285 (+ 954us) server_negotiation.cc:407] Connection header received
0619 02:58:37.251530 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.251535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.251604 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.251691 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:58:37.252296 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.253365 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.256296 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.257536 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.257680 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.257742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.258276 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.258404 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:58:37.258651 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52329b2f52ac41b9b812da479ec4c579" candidate_uuid: "0912ca04c396413ba167f6091d52c5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd82d4878dd4478aa54050ed4f49101" is_pre_election: true
I20260619 02:58:37.259232 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.249944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:43107 (local address 127.22.230.2:37603)
0619 02:58:37.251190 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0619 02:58:37.251207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:58:37.251370 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:58:37.251721 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:58:37.251725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:58:37.251748 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:58:37.252124 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.252134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.253657 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.253662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.256145 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:58:37.256160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.257918 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.257925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:58:37.258094 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.258125 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:58:37.258835 (+ 710us) client_negotiation.cc:240] Negotiation successful
0619 02:58:37.258952 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"thread_start_us":786,"threads_started":1}
I20260619 02:58:37.259214 2474 raft_consensus.cc:2454] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0912ca04c396413ba167f6091d52c5bc in term 0.
I20260619 02:58:37.259852 2545 leader_election.cc:302] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0912ca04c396413ba167f6091d52c5bc, dbd82d4878dd4478aa54050ed4f49101; no voters:
I20260619 02:58:37.260207 2821 raft_consensus.cc:2781] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:58:37.260344 2821 raft_consensus.cc:482] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:58:37.260433 2821 raft_consensus.cc:3037] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:58:37.260591 2751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52329b2f52ac41b9b812da479ec4c579" candidate_uuid: "0912ca04c396413ba167f6091d52c5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae42342a8ec42a4aad50074983a5aef" is_pre_election: true
I20260619 02:58:37.261011 2751 raft_consensus.cc:2454] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0912ca04c396413ba167f6091d52c5bc in term 0.
I20260619 02:58:37.263429 2821 raft_consensus.cc:504] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
I20260619 02:58:37.264074 2821 leader_election.cc:288] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [CANDIDATE]: Term 1 election: Requested vote from peers dbd82d4878dd4478aa54050ed4f49101 (127.22.230.1:35597), aae42342a8ec42a4aad50074983a5aef (127.22.230.3:43107)
I20260619 02:58:37.264420 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52329b2f52ac41b9b812da479ec4c579" candidate_uuid: "0912ca04c396413ba167f6091d52c5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd82d4878dd4478aa54050ed4f49101"
I20260619 02:58:37.264482 2751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52329b2f52ac41b9b812da479ec4c579" candidate_uuid: "0912ca04c396413ba167f6091d52c5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aae42342a8ec42a4aad50074983a5aef"
I20260619 02:58:37.264595 2474 raft_consensus.cc:3037] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:58:37.264673 2751 raft_consensus.cc:3037] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:58:37.267670 2474 raft_consensus.cc:2454] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0912ca04c396413ba167f6091d52c5bc in term 1.
I20260619 02:58:37.267809 2751 raft_consensus.cc:2454] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0912ca04c396413ba167f6091d52c5bc in term 1.
I20260619 02:58:37.268064 2545 leader_election.cc:302] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0912ca04c396413ba167f6091d52c5bc, dbd82d4878dd4478aa54050ed4f49101; no voters:
I20260619 02:58:37.268452 2821 raft_consensus.cc:2781] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:58:37.268895 2821 raft_consensus.cc:686] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 1 LEADER]: Becoming Leader. State: Replica: 0912ca04c396413ba167f6091d52c5bc, State: Running, Role: LEADER
I20260619 02:58:37.269393 2821 consensus_queue.cc:237] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } }
W20260619 02:58:37.271159 2797 tablet.cc:2369] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:58:37.274652 2306 catalog_manager.cc:5495] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0912ca04c396413ba167f6091d52c5bc (127.22.230.2). New cstate: current_term: 1 leader_uuid: "0912ca04c396413ba167f6091d52c5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0912ca04c396413ba167f6091d52c5bc" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 40083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 } health_report { overall_health: UNKNOWN } } }
I20260619 02:58:37.296204 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.290322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.22.230.2:40083)
0619 02:58:37.290524 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.290530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.290548 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:58:37.290741 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.290747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.290834 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.290972 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:58:37.291329 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.292467 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.293933 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.294545 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.294693 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.295128 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:58:37.295713 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:58:37.295843 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.295883 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.295954 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:58:37.305365 2474 raft_consensus.cc:1261] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 1 FOLLOWER]: Refusing update from remote peer 0912ca04c396413ba167f6091d52c5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:58:37.305359 2751 raft_consensus.cc:1261] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 1 FOLLOWER]: Refusing update from remote peer 0912ca04c396413ba167f6091d52c5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:58:37.306354 2826 consensus_queue.cc:1035] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aae42342a8ec42a4aad50074983a5aef" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:58:37.306763 2821 consensus_queue.cc:1035] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd82d4878dd4478aa54050ed4f49101" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 35597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:58:37.332451 2305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:58:37.332681 2305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:58:37.336643 2305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be4aafc284da46f98efd9358cd002ff9: Sending DeleteTablet for 3 replicas of tablet 52329b2f52ac41b9b812da479ec4c579
I20260619 02:58:37.337731 2597 tablet_service.cc:1514] Processing DeleteTablet for tablet 52329b2f52ac41b9b812da479ec4c579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:58:37 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:49982
I20260619 02:58:37.337818 2454 tablet_service.cc:1514] Processing DeleteTablet for tablet 52329b2f52ac41b9b812da479ec4c579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:58:37 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:43734
I20260619 02:58:37.337821 2731 tablet_service.cc:1514] Processing DeleteTablet for tablet 52329b2f52ac41b9b812da479ec4c579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:58:37 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:33348
I20260619 02:58:37.338521 2839 tablet_replica.cc:330] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: stopping tablet replica
I20260619 02:58:37.338949 2839 raft_consensus.cc:2229] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:58:37.339597 2839 raft_consensus.cc:2258] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:58:37.340164 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 2386
I20260619 02:58:37.341495 2839 ts_tablet_manager.cc:1905] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:58:37.348899 2840 tablet_replica.cc:330] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: stopping tablet replica
I20260619 02:58:37.350296 2840 raft_consensus.cc:2229] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:58:37.350878 2839 ts_tablet_manager.cc:1918] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:58:37.351076 2839 log.cc:1196] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/52329b2f52ac41b9b812da479ec4c579
I20260619 02:58:37.351346 2840 raft_consensus.cc:2258] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:58:37.351558 2839 ts_tablet_manager.cc:1939] T 52329b2f52ac41b9b812da479ec4c579 P 0912ca04c396413ba167f6091d52c5bc: Deleting consensus metadata
I20260619 02:58:37.352684 2841 tablet_replica.cc:330] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: stopping tablet replica
I20260619 02:58:37.353336 2291 catalog_manager.cc:4841] TS 0912ca04c396413ba167f6091d52c5bc (127.22.230.2:40083): tablet 52329b2f52ac41b9b812da479ec4c579 (table test-table [id=4f230c39bc1347c0b7d6a7e4d67c663d]) successfully deleted
I20260619 02:58:37.354447 2840 ts_tablet_manager.cc:1905] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 02:58:37 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:58:37.361783 2840 ts_tablet_manager.cc:1918] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:58:37.361949 2840 log.cc:1196] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/52329b2f52ac41b9b812da479ec4c579
I20260619 02:58:37.362498 2840 ts_tablet_manager.cc:1939] T 52329b2f52ac41b9b812da479ec4c579 P dbd82d4878dd4478aa54050ed4f49101: Deleting consensus metadata
I20260619 02:58:37.362999 2841 raft_consensus.cc:2229] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:58:37.363479 2841 raft_consensus.cc:2258] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:58:37.365597 2290 catalog_manager.cc:4841] TS dbd82d4878dd4478aa54050ed4f49101 (127.22.230.1:35597): tablet 52329b2f52ac41b9b812da479ec4c579 (table test-table [id=4f230c39bc1347c0b7d6a7e4d67c663d]) successfully deleted
I20260619 02:58:37.365885 2841 ts_tablet_manager.cc:1905] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:58:37.367733 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.341383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.22.230.1:35597)
0619 02:58:37.341608 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.341617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.351151 (+ 9534us) server_negotiation.cc:407] Connection header received
0619 02:58:37.351457 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.351462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.351559 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.351665 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:37.353924 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.354721 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.356158 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.356649 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.356797 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.360707 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:37.360718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:37.360731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:37.360773 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:37.364078 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.364976 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.364985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.364989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.365155 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.365647 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.365651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.365654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.366161 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:37.366419 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.367123 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.367379 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:58:37.368758 2514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:58:37.371243 2841 ts_tablet_manager.cc:1918] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:58:37.371429 2841 log.cc:1196] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/52329b2f52ac41b9b812da479ec4c579
I20260619 02:58:37.371987 2841 ts_tablet_manager.cc:1939] T 52329b2f52ac41b9b812da479ec4c579 P aae42342a8ec42a4aad50074983a5aef: Deleting consensus metadata
I20260619 02:58:37.373382 2290 catalog_manager.cc:4841] TS aae42342a8ec42a4aad50074983a5aef (127.22.230.3:43107): tablet 52329b2f52ac41b9b812da479ec4c579 (table test-table [id=4f230c39bc1347c0b7d6a7e4d67c663d]) successfully deleted
I20260619 02:58:37.459825 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 2386
I20260619 02:58:37.482981 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 2525
Jun 19 02:58:37 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:58:37.500154 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.483861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.22.230.2:40083)
0619 02:58:37.484022 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.484026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.484043 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:58:37.484167 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.484171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.484229 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.484312 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:37.485843 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.486884 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.488755 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.489141 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.489227 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.494259 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:37.494269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:37.494283 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:37.494316 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:37.497437 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.498049 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.498061 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.498063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.498206 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.498613 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.498620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.498626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.498975 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:37.499321 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.499615 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.499819 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 02:58:37.500885 2657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:58:37.606884 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 2525
I20260619 02:58:37.624449 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 2668
Jun 19 02:58:37 dist-test-slave-2gms krb5kdc[1830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:58:37.638415 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:58:37.625247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.22.230.3:43107)
0619 02:58:37.625420 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 02:58:37.625424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:58:37.625442 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:58:37.625551 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:58:37.625555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:58:37.625615 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:58:37.625693 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:58:37.627003 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.627708 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.628909 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:58:37.629252 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:58:37.629339 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:58:37.633328 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:58:37.633339 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:58:37.633369 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:58:37.633401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:58:37.636249 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.636788 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.636794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.636796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.636896 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:58:37.637181 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:58:37.637184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:58:37.637186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:58:37.637518 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:58:37.637713 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:58:37.637917 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 02:58:37.638056 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 02:58:37.639196 2791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:58:37.725867 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 2668
I20260619 02:58:37.741701 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 2274
I20260619 02:58:37.743106 2336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:58:37.851150 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 2274
I20260619 02:58:37.875509 23448 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:58:38.210004 23448 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:58:38.210189 23448 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:58:38.554787 23448 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:58:38.555 UTC [1860] LOG: received smart shutdown request
2026-06-19 02:58:38.558 UTC [1860] LOG: background worker "logical replication launcher" (PID 1869) exited with exit code 1
2026-06-19 02:58:38.558 UTC [1864] LOG: shutting down
2026-06-19 02:58:38.701 UTC [1860] LOG: database system is shut down
2026-06-19T02:58:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46382 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:58:38 dist-test-slave-2gms krb5kdc[2855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:58:38 dist-test-slave-2gms krb5kdc[2855](info): set up 2 sockets
Jun 19 02:58:38 dist-test-slave-2gms krb5kdc[2855](info): commencing operation
krb5kdc: starting...
W20260619 02:58:39.923123 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:58:40 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:58:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/rangeradmin_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/rangeradmin_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/rangerlookup_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/rangerlookup_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
I20260619 02:58:40.231328 23448 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrIFYxA/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/postgres -l logfile start
2026-06-19 02:58:42.546 UTC [2885] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 02:58:42.546 UTC [2885] LOG: listening on IPv4 address "127.22.230.20", port 34765
2026-06-19 02:58:42.549 UTC [2885] LOG: listening on Unix socket "/tmp/.s.PGSQL.34765"
2026-06-19 02:58:42.568 UTC [2888] LOG: database system was shut down at 2026-06-19 02:58:41 UTC
2026-06-19 02:58:42.572 UTC [2885] LOG: database system is ready to accept connections
I20260619 02:58:43.514708 23448 mini_postgres.cc:96] Postgres bound to 34765
2026-06-19 02:58:43.532 UTC [2896] FATAL: database "slave" does not exist
127.22.230.20:34765 - accepting connections
I20260619 02:58:43.533004 23448 mini_ranger.cc:161] Starting Ranger...
I20260619 02:58:43.557792 23448 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 02:58:43.980156 23448 mini_ranger.cc:87] Created ranger Postgres database
I20260619 02:58:43.980243 23448 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-admin
2026-06-19 02:58:44,265 [I] DB FLAVOR :POSTGRES
2026-06-19 02:58:44,265 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:58:44,266 [I] Checking connection..
2026-06-19 02:58:44,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 02:58:44,502 [I] Checking connection passed.
2026-06-19 02:58:44,502 [I] --------- Verifying version history table ---------
2026-06-19 02:58:44,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:58:44,739 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:58:44,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:58:44,978 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 02:58:44,978 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 02:58:44,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 02:58:45.216 UTC [2960] WARNING: there is no transaction in progress
2026-06-19 02:58:45,222 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 02:58:45,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 02:58:45,477 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 02:58:45,477 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:58:45,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 02:58:45,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 02:58:45,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2gms.c.gcp-upstream.internal','N') ;"
2026-06-19 02:58:46,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:58:46,418 [I] Table x_portal_user does not exist in database ranger
2026-06-19 02:58:46,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:58:46,659 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 02:58:46,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:58:46,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:58:47,149 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 02:58:47,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 02:58:48.006 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.022 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.038 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.052 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.068 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.084 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.134 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.140 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.153 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.162 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.171 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.185 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.194 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.199 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.205 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48.211 UTC [3068] WARNING: there is no transaction in progress
2026-06-19 02:58:48,760 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 02:58:48,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 02:58:49,007 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 02:58:49,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 02:58:49,261 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 02:58:49,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 02:58:49,509 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 02:58:49,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 02:58:49,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 02:58:50,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2gms.c.gcp-upstream.internal';"
2026-06-19 02:58:50,212 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 02:58:50,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 02:58:50,427 [I] DB_PATCHES have already been applied
I20260619 02:58:50.433501 23448 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrIFYxA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 02:58:50.433583 23448 mini_ranger.cc:193] Using host: 127.22.230.20
I20260619 02:58:50.448855 23448 mini_ranger.cc:227] Ranger admin URL: http://127.22.230.20:41629
Jun 19, 2026 2:58:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:58:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:58:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:58:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:58:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:58:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:58:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 02:58:51.574443 23448 mini_ranger.cc:160] Time spent starting Ranger: real 8.041s user 0.000s sys 0.039s
Jun 19, 2026 2:59:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:59:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:59:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:59:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:59:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 02:59:12.646725 23448 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/rangerkms_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/rangerkms_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.230.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.230.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.230.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/HTTP_127.22.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 02:59:12 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837952, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 02:59:12.801482 23448 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 02:59:12.821480 23448 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 02:59:13.547399 23448 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 02:59:13.547518 23448 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms
I20260619 02:59:13.547564 23448 mini_ranger_kms.cc:227] Using postgres at 127.22.230.20:34765
2026-06-19 02:59:13,894 [I] DB FLAVOR :POSTGRES
2026-06-19 02:59:13,894 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 02:59:13,894 [I] Checking connection
2026-06-19 02:59:13,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 02:59:14,117 [I] connection success
2026-06-19 02:59:14,117 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 02:59:14,117 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 02:59:14,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 02:59:14,332 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 02:59:14,332 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 02:59:14,332 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 02:59:14,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.230.20:34765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 02:59:14,583 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 02:59:15.027813 23448 mini_ranger_kms.cc:310] Created kms service
I20260619 02:59:15.226430 23448 mini_ranger_kms.cc:326] Created kudu user
I20260619 02:59:15.269213 23448 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 02:59:15 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837952, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:15.613766 23448 mini_ranger_kms.cc:384] Added ranger policy
I20260619 02:59:15.614112 23448 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrIFYxA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrIFYxA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 02:59:15.614174 23448 mini_ranger_kms.cc:240] Using host: 127.22.230.20
I20260619 02:59:15.631989 23448 mini_ranger_kms.cc:276] Ranger KMS PID: 3248
I20260619 02:59:15.632114 23448 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.230.20:37481
Jun 19, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:59:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:59:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 02:59:17 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.230.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:17.843010 23448 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.042s user 0.086s sys 0.280s
I20260619 02:59:17.843251 23448 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 02:59:17.843458 23448 mini_ranger_kms.cc:401] 127.22.230.20:37481/kms/v1/keys
Jun 19 02:59:18 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.230.20@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
Jun 19 02:59:18 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.230.20@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
Jun 19 02:59:20 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:59:20.875100 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33263
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:33263
--ranger_config_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:21.193301 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:21.193607 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:21.193779 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:21.200654 3299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:21.200785 3299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:21.200911 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:21.201011 3299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:21.201107 3299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:21.210412 3299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.230.20:37481/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:33263
--ranger_config_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:21.211568 3299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:21.212699 3299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:21.223976 3305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:21.223973 3304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:21.224042 3307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:21.224746 3299 server_base.cc:1034] running on GCE node
I20260619 02:59:21.225504 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:21.226794 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:21.228003 3299 hybrid_clock.cc:648] HybridClock initialized: now 1781837961227966 us; error 30 us; skew 500 ppm
I20260619 02:59:21.228334 3299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:21 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837961, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:21.231251 3299 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:21.232556 3299 webserver.cc:460] Webserver started at http://127.22.230.62:38701/ using document root <none> and password file <none>
I20260619 02:59:21.233093 3299 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:21.233177 3299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:21.233395 3299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:59:21 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837961, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:21.287770 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "4e2eb497bbb14af6a7cfd63a81f793c9"
format_stamp: "Formatted at 2026-06-19 02:59:21 on dist-test-slave-2gms"
server_key: "e5f6dc2963e9b270c51c8cecb4a1166a"
server_key_iv: "e8ca4436c682674b08d532666d9a3bd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:21.288635 3299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "4e2eb497bbb14af6a7cfd63a81f793c9"
format_stamp: "Formatted at 2026-06-19 02:59:21 on dist-test-slave-2gms"
server_key: "e5f6dc2963e9b270c51c8cecb4a1166a"
server_key_iv: "e8ca4436c682674b08d532666d9a3bd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:21.293707 3299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:59:21.310927 3314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:21.312245 3299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:59:21.312435 3299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "4e2eb497bbb14af6a7cfd63a81f793c9"
format_stamp: "Formatted at 2026-06-19 02:59:21 on dist-test-slave-2gms"
server_key: "e5f6dc2963e9b270c51c8cecb4a1166a"
server_key_iv: "e8ca4436c682674b08d532666d9a3bd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:21.312652 3299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:21.329697 3299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:21.333083 3299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:21.333406 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:21.351467 3299 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:33263
I20260619 02:59:21.351488 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:33263 every 8 connection(s)
I20260619 02:59:21.352663 3299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:21.356066 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:21.358120 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 3299
I20260619 02:59:21.358325 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:59:21.359161 23448 external_mini_cluster.cc:1428] Setting key cfdcf60349c3985aef36a6c69e8b3c40
I20260619 02:59:21.368229 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Bootstrap starting.
I20260619 02:59:21.371820 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:21.373018 3366 log.cc:824] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:59:21 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:21.375730 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: No bootstrap required, opened a new log
I20260619 02:59:21.381383 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:21.361049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.22.230.62:33263)
0619 02:59:21.361912 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:21.361923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:21.361962 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:59:21.363698 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:21.363736 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:21.364080 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:21.364366 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:21.365941 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:21.367319 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:21.369145 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:21.369792 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:21.369931 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:21.374685 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:21.374694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:21.374706 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:21.374742 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:21.377682 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:21.378331 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:21.378345 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:21.378350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:21.378489 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:21.378855 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:21.378859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:21.378862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:21.379334 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:21.379666 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:21.380136 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 02:59:21.380464 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":447,"threads_started":1}
I20260619 02:59:21.383249 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:21.383599 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:21.383716 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2eb497bbb14af6a7cfd63a81f793c9, State: Initialized, Role: FOLLOWER
I20260619 02:59:21.384253 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:21.384473 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:59:21.384632 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:59:21.384789 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:21.386796 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:21.387287 3366 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2eb497bbb14af6a7cfd63a81f793c9; no voters:
I20260619 02:59:21.387812 3366 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:59:21.387990 3371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:21.389149 3371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e2eb497bbb14af6a7cfd63a81f793c9, State: Running, Role: LEADER
I20260619 02:59:21.389714 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:21.389672 3371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:21.392581 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } } }
I20260619 02:59:21.392573 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2eb497bbb14af6a7cfd63a81f793c9. Latest consensus state: current_term: 1 leader_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } } }
I20260619 02:59:21.393006 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:21.393006 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:21.393810 3380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:21.398816 3380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:21.402629 3366 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 02:59:21.414952 3380 catalog_manager.cc:1347] Generated new cluster ID: aef0c01218a9414ba1be166d96f20c77
I20260619 02:59:21.415107 3380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:59:21.467887 3380 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:59:21.469753 3380 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:59:21.479509 3380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Generated new TSK 0
I20260619 02:59:21.480429 3380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:59:22.159595 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33263
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:59:22 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837962, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 02:59:22.580845 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:22.581259 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:22.581482 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:22.590899 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:22.591039 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:22.591259 3411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:59:22.603029 3411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.230.20:37481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:22.604560 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:22.605965 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:22.628110 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:22.629050 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:22.636144 3411 server_base.cc:1034] running on GCE node
W20260619 02:59:22.635958 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:22.637163 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:22.638406 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:22.639648 3411 hybrid_clock.cc:648] HybridClock initialized: now 1781837962639581 us; error 64 us; skew 500 ppm
I20260619 02:59:22.639990 3411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:22 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837962, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:22.643102 3411 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:59:22.644925 3411 webserver.cc:460] Webserver started at http://127.22.230.1:43155/ using document root <none> and password file <none>
I20260619 02:59:22.645622 3411 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:22.645759 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:22.646062 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:59:22 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837962, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:22.666464 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "9e41534198c6455da33c08dd8356cc85"
format_stamp: "Formatted at 2026-06-19 02:59:22 on dist-test-slave-2gms"
server_key: "7afec98b33604566dcf55a38abddf610"
server_key_iv: "d2928fa25d6f2324ff2a6ebc52db4152"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:22.667701 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "9e41534198c6455da33c08dd8356cc85"
format_stamp: "Formatted at 2026-06-19 02:59:22 on dist-test-slave-2gms"
server_key: "7afec98b33604566dcf55a38abddf610"
server_key_iv: "d2928fa25d6f2324ff2a6ebc52db4152"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:22.683836 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.006s sys 0.008s
I20260619 02:59:22.721408 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:22.723316 3411 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.009s
I20260619 02:59:22.723580 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "9e41534198c6455da33c08dd8356cc85"
format_stamp: "Formatted at 2026-06-19 02:59:22 on dist-test-slave-2gms"
server_key: "7afec98b33604566dcf55a38abddf610"
server_key_iv: "d2928fa25d6f2324ff2a6ebc52db4152"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:22.723894 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:22.743304 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:22.747257 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:22.747591 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:22.748621 3411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:22.749979 3411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:22.750077 3411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:22.750196 3411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:22.750284 3411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:22.859508 3411 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:41433
I20260619 02:59:22.859546 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:41433 every 8 connection(s)
I20260619 02:59:22.861155 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:59:22.869590 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 3411
I20260619 02:59:22.869818 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:59:22.870599 23448 external_mini_cluster.cc:1428] Setting key 50d4e3a1194a6f4cf6df701281f7dc3a
Jun 19 02:59:22 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837962, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:22.890241 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:22.865959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:36797 (local address 127.22.230.62:33263)
0619 02:59:22.869378 (+ 3419us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:22.869384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:22.872524 (+ 3140us) server_negotiation.cc:407] Connection header received
0619 02:59:22.872583 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:22.872588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:22.872673 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:22.872775 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:22.874794 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:22.875907 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:22.878224 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:22.878876 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:22.878967 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:22.883116 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:22.883122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:22.883126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:22.883171 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:22.886638 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:22.887560 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:22.887565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:22.887568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:22.887705 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:22.888269 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:22.888274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:22.888277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:22.888620 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:22.888769 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:22.889667 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 02:59:22.889878 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3278,"thread_start_us":3180,"threads_started":1}
I20260619 02:59:22.892570 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:22.864528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33263 (local address 127.22.230.1:36797)
0619 02:59:22.869865 (+ 5337us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:22.869924 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:22.872180 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:22.873181 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:22.873192 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:22.873786 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:22.874580 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:22.874610 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:22.876105 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:22.876111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:22.878061 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:22.878074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:22.879285 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:22.879304 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:22.879558 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:22.880313 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:22.880341 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:22.882859 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:22.886854 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:22.886863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:22.886879 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:22.887425 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:22.887850 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:22.887855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:22.887858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:22.888132 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:22.888852 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:22.888861 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:22.889244 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 02:59:22.890757 (+ 1513us) client_negotiation.cc:240] Negotiation successful
0619 02:59:22.891234 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5058,"thread_start_us":5125,"threads_started":1}
I20260619 02:59:22.894716 3544 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33263
I20260619 02:59:22.895007 3544 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:22.895821 3544 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:22.898267 3331 ts_manager.cc:194] Registered new tserver with Master: 9e41534198c6455da33c08dd8356cc85 (127.22.230.1:41433)
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
I20260619 02:59:22.900221 3331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:36797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:59:22.941579 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33263
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:59:22 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837962, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
Jun 19 02:59:23 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837962, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
W20260619 02:59:23.401111 3550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:23.401602 3550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:23.401795 3550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:23.412052 3550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:23.412201 3550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:23.412408 3550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:59:23.422536 3550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.230.20:37481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:23.423821 3550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:23.425081 3550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:23.444744 3558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:23.444741 3557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:23.452386 3550 server_base.cc:1034] running on GCE node
W20260619 02:59:23.452737 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:23.453500 3550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:23.461915 3550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:23.463182 3550 hybrid_clock.cc:648] HybridClock initialized: now 1781837963463126 us; error 67 us; skew 500 ppm
I20260619 02:59:23.463538 3550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:23 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:23.466981 3550 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:59:23.468845 3550 webserver.cc:460] Webserver started at http://127.22.230.2:41819/ using document root <none> and password file <none>
I20260619 02:59:23.469570 3550 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:23.469715 3550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:23.470036 3550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:59:23 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:23.485427 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "8d5ce0ac83454bbbb7e4954587b8498f"
format_stamp: "Formatted at 2026-06-19 02:59:23 on dist-test-slave-2gms"
server_key: "096c9555adec0a437addbc28a90824d4"
server_key_iv: "5f12839dec5627f15681c9348d2aac66"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:23.486507 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "8d5ce0ac83454bbbb7e4954587b8498f"
format_stamp: "Formatted at 2026-06-19 02:59:23 on dist-test-slave-2gms"
server_key: "096c9555adec0a437addbc28a90824d4"
server_key_iv: "5f12839dec5627f15681c9348d2aac66"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:23.493331 3550 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 02:59:23.510131 3574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:23.511848 3550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:59:23.512005 3550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "8d5ce0ac83454bbbb7e4954587b8498f"
format_stamp: "Formatted at 2026-06-19 02:59:23 on dist-test-slave-2gms"
server_key: "096c9555adec0a437addbc28a90824d4"
server_key_iv: "5f12839dec5627f15681c9348d2aac66"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:23.512223 3550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:23.542066 3550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:23.546934 3550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:23.547277 3550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:23.548394 3550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:23.550029 3550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:23.550127 3550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:23.550242 3550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:23.550323 3550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:23.589650 3550 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:43385
I20260619 02:59:23.589696 3686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:43385 every 8 connection(s)
I20260619 02:59:23.591312 3550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:59:23.591918 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 3550
I20260619 02:59:23.592085 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:59:23.592660 23448 external_mini_cluster.cc:1428] Setting key 2346bf7f87c6206950f7960283220efe
Jun 19 02:59:23 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837963, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
I20260619 02:59:23.635433 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:23.601837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:60763 (local address 127.22.230.62:33263)
0619 02:59:23.607556 (+ 5719us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:23.607563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:23.613738 (+ 6175us) server_negotiation.cc:407] Connection header received
0619 02:59:23.613813 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:23.613819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:23.613920 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:23.614033 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:23.615852 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:23.617208 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:23.622694 (+ 5486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:23.623280 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:23.623406 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:23.628028 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:23.628035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:23.628040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:23.628077 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:23.631474 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:23.632362 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:23.632367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:23.632371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:23.632507 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:23.632995 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:23.633000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:23.633003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:23.633329 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:23.633460 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:23.634822 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0619 02:59:23.635052 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5591,"thread_start_us":5495,"threads_started":1}
I20260619 02:59:23.636368 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:23.601317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33263 (local address 127.22.230.2:60763)
0619 02:59:23.611052 (+ 9735us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:23.611107 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:23.613393 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:23.614336 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:23.614347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:23.614924 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:23.615662 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:23.615684 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:23.617368 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:23.617373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:23.619400 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:23.619414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:23.623676 (+ 4262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:23.623683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:23.623951 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:23.624725 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:23.624759 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:23.627728 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:23.631659 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:23.631667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:23.631684 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:23.632223 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:23.632623 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:23.632627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:23.632631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:23.632862 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:23.633538 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:23.633545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:23.633911 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 02:59:23.634217 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 02:59:23.634727 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9435,"thread_start_us":9209,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
I20260619 02:59:23.638125 3687 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33263
I20260619 02:59:23.638463 3687 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:23.639273 3687 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:23.643340 3331 ts_manager.cc:194] Registered new tserver with Master: 8d5ce0ac83454bbbb7e4954587b8498f (127.22.230.2:43385)
I20260619 02:59:23.644367 3331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:60763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:59:23.672250 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33263
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:59:23.903664 3544 heartbeater.cc:503] Master 127.22.230.62:33263 was elected leader, sending a full tablet report...
W20260619 02:59:24.026232 3693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:24.026546 3693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:24.026732 3693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:24.033617 3693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:24.033751 3693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:24.033936 3693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:59:24.043041 3693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.230.20:37481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:24.044596 3693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:24.045749 3693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:24.058851 3698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:24.058895 3701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:24.058812 3699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:24.062131 3693 server_base.cc:1034] running on GCE node
I20260619 02:59:24.062847 3693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:24.063954 3693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:24.065171 3693 hybrid_clock.cc:648] HybridClock initialized: now 1781837964065108 us; error 45 us; skew 500 ppm
I20260619 02:59:24.065519 3693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:24 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:24.069173 3693 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:59:24.070483 3693 webserver.cc:460] Webserver started at http://127.22.230.3:41295/ using document root <none> and password file <none>
I20260619 02:59:24.071153 3693 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:24.071305 3693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:24.071620 3693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 02:59:24 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:24.088162 3693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "2fdec5fe2ec84a5396e0a2e204078f9d"
format_stamp: "Formatted at 2026-06-19 02:59:24 on dist-test-slave-2gms"
server_key: "224d3145ba4e6956201a2e2cd93fddaa"
server_key_iv: "9fd86a76ca02d0520a40493c48de8cff"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:24.088937 3693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "2fdec5fe2ec84a5396e0a2e204078f9d"
format_stamp: "Formatted at 2026-06-19 02:59:24 on dist-test-slave-2gms"
server_key: "224d3145ba4e6956201a2e2cd93fddaa"
server_key_iv: "9fd86a76ca02d0520a40493c48de8cff"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:24.094406 3693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 02:59:24.108911 3708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:24.110435 3693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 02:59:24.110679 3693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "2fdec5fe2ec84a5396e0a2e204078f9d"
format_stamp: "Formatted at 2026-06-19 02:59:24 on dist-test-slave-2gms"
server_key: "224d3145ba4e6956201a2e2cd93fddaa"
server_key_iv: "9fd86a76ca02d0520a40493c48de8cff"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:24.110921 3693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:24.128479 3693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:24.132180 3693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:24.132468 3693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:24.133353 3693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:24.134608 3693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:24.134716 3693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:24.134868 3693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:24.134953 3693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:24.166656 3693 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:45681
I20260619 02:59:24.166698 3820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:45681 every 8 connection(s)
I20260619 02:59:24.167881 3693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:59:24.170661 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 3693
I20260619 02:59:24.170822 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:59:24.171434 23448 external_mini_cluster.cc:1428] Setting key 08671b6f9064437c0a300406f315f780
Jun 19 02:59:24 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837964, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:24.187001 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:24.170336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:54599 (local address 127.22.230.62:33263)
0619 02:59:24.170932 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:24.170938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:24.171847 (+ 909us) server_negotiation.cc:407] Connection header received
0619 02:59:24.174020 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:24.174025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:24.174105 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:24.174258 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:24.175612 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.176314 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.177841 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.178236 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.178332 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:24.181522 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:24.181527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:24.181530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:24.181567 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:24.184012 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.184626 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.184630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.184632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.184745 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.185230 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.185236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.185240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.185631 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:24.185735 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:24.186533 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 02:59:24.186748 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":275,"threads_started":1}
I20260619 02:59:24.187882 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:24.170854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33263 (local address 127.22.230.3:54599)
0619 02:59:24.171664 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:24.171711 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:24.173784 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:24.174536 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:24.174548 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:24.174964 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:24.175470 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:24.175495 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.176443 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.176447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:24.177606 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:24.177616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.178420 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.178424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:24.178690 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:24.179383 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:24.179403 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:24.181348 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:24.184129 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:24.184134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:24.184144 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:24.184523 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:24.184856 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:24.184861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:24.184864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:24.185073 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:24.185757 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:24.185765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:24.186134 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 02:59:24.186382 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 02:59:24.186891 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":316,"threads_started":1}
I20260619 02:59:24.189144 3821 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33263
I20260619 02:59:24.189380 3821 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:24.189925 3821 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:24.191313 3331 ts_manager.cc:194] Registered new tserver with Master: 2fdec5fe2ec84a5396e0a2e204078f9d (127.22.230.3:45681)
I20260619 02:59:24.192056 3331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:54599
I20260619 02:59:24.198400 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:59:24.200098 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 3411
Jun 19 02:59:24 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:59:24.215827 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:24.200957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.22.230.1:41433)
0619 02:59:24.201541 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:24.201546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:24.201568 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:59:24.201616 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:24.201621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:24.201775 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:24.201887 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:24.203246 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.204223 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.205638 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.206112 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.206200 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:24.210193 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:24.210203 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:24.210214 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:24.210242 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:24.212991 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.213687 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.213693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.213695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.213845 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.214219 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.214225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.214228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.214562 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:24.214951 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:24.215240 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:59:24.215492 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":298,"threads_started":1}
I20260619 02:59:24.216976 3539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:24.297652 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 3411
I20260619 02:59:24.313764 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 3550
Jun 19 02:59:24 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:59:24.330241 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:24.314710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.22.230.2:43385)
0619 02:59:24.315283 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:24.315290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:24.315314 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:59:24.315373 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:24.315379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:24.315593 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:24.315769 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:24.317086 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.318106 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.319629 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.320156 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.320307 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:24.324422 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:24.324432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:24.324444 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:24.324475 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:24.327778 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.328533 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.328539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.328541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.328657 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.328978 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.328983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.328985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.329279 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:24.329503 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:24.329716 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 02:59:24.329939 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":306,"threads_started":1}
I20260619 02:59:24.331496 3682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:24.411783 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 3550
I20260619 02:59:24.431718 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 3693
Jun 19 02:59:24 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:59:24.446666 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:24.432617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.22.230.3:45681)
0619 02:59:24.433197 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:24.433203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:24.433231 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:59:24.433285 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:24.433291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:24.433498 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:24.433614 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:24.434905 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.435872 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.437054 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:24.437583 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:24.437675 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:24.441654 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:24.441663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:24.441675 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:24.441701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:24.444494 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.445047 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.445053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.445055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.445162 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:24.445444 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:24.445447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:24.445448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:24.445726 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:24.445936 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:24.446141 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 02:59:24.446308 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":301,"threads_started":1}
I20260619 02:59:24.447847 3816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:24.528738 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 3693
I20260619 02:59:24.544384 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 3299
I20260619 02:59:24.545663 3361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:24.650326 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 3299
I20260619 02:59:24.675606 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33263
--webserver_interface=127.22.230.62
--webserver_port=38701
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:33263
--ranger_config_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:24.975747 3833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:24.975989 3833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:24.976135 3833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:24.982669 3833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:24.982784 3833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:24.982889 3833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:24.982986 3833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:24.983074 3833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:24.992431 3833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:33263
--ranger_config_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=38701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:24.993465 3833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:24.994681 3833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:25.009781 3839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:25.009806 3838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:25.009888 3841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:25.010775 3833 server_base.cc:1034] running on GCE node
I20260619 02:59:25.011533 3833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:25.012779 3833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:25.014009 3833 hybrid_clock.cc:648] HybridClock initialized: now 1781837965013956 us; error 45 us; skew 500 ppm
I20260619 02:59:25.014353 3833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:25 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:25.017288 3833 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:25.018608 3833 webserver.cc:460] Webserver started at http://127.22.230.62:38701/ using document root <none> and password file <none>
I20260619 02:59:25.019202 3833 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:25.019348 3833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:59:25 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:25.068136 3833 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:59:25.070951 3849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:25.072234 3833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:59:25.072423 3833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "4e2eb497bbb14af6a7cfd63a81f793c9"
format_stamp: "Formatted at 2026-06-19 02:59:21 on dist-test-slave-2gms"
server_key: "e5f6dc2963e9b270c51c8cecb4a1166a"
server_key_iv: "e8ca4436c682674b08d532666d9a3bd9"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:25.073364 3833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:25.091320 3833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:25.094681 3833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:25.094987 3833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:25.113164 3833 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:33263
I20260619 02:59:25.113147 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:33263 every 8 connection(s)
I20260619 02:59:25.114593 3833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:25.116267 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 3833
I20260619 02:59:25.117333 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:41433
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:33263
--webserver_port=43155
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:59:25.123991 3901 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:59:25.127959 3901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Bootstrap starting.
I20260619 02:59:25.145843 3901 log.cc:824] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:25.153859 3901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:59:25.154465 3901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Bootstrap complete.
I20260619 02:59:25.163147 3901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:25.163892 3901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2eb497bbb14af6a7cfd63a81f793c9, State: Initialized, Role: FOLLOWER
I20260619 02:59:25.164609 3901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:25.164831 3901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:59:25.164937 3901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:59:25.165122 3901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:59:25.168659 3901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:25.169121 3901 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2eb497bbb14af6a7cfd63a81f793c9; no voters:
I20260619 02:59:25.169742 3901 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:59:25.169862 3905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:59:25.170909 3905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [term 2 LEADER]: Becoming Leader. State: Replica: 4e2eb497bbb14af6a7cfd63a81f793c9, State: Running, Role: LEADER
I20260619 02:59:25.171484 3905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } }
I20260619 02:59:25.171891 3901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:25.173923 3907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2eb497bbb14af6a7cfd63a81f793c9. Latest consensus state: current_term: 2 leader_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } } }
I20260619 02:59:25.174345 3907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:25.174826 3906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2eb497bbb14af6a7cfd63a81f793c9" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33263 } } }
I20260619 02:59:25.175158 3906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:25.178685 3912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:25.190416 3912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:25.191738 3912 catalog_manager.cc:1259] Loaded cluster ID: aef0c01218a9414ba1be166d96f20c77
I20260619 02:59:25.191838 3912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:59:25.195672 3912 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:59:25.197036 3912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Loaded TSK: 0
I20260619 02:59:25.198463 3912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:59:25.204074 3901 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 02:59:25.555156 3903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:25.555601 3903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:25.555809 3903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:25.567970 3903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:25.568109 3903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:25.568356 3903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:59:25.583669 3903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:41433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=43155
--tserver_master_addrs=127.22.230.62:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:25.585140 3903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:25.586855 3903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:25.614684 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:25.616052 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:25.621090 3903 server_base.cc:1034] running on GCE node
W20260619 02:59:25.621737 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:25.622627 3903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:25.630030 3903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:25.631325 3903 hybrid_clock.cc:648] HybridClock initialized: now 1781837965631254 us; error 59 us; skew 500 ppm
I20260619 02:59:25.631721 3903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:25 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:25.635974 3903 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:59:25.645538 3903 webserver.cc:460] Webserver started at http://127.22.230.1:43155/ using document root <none> and password file <none>
I20260619 02:59:25.646446 3903 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:25.646692 3903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:59:25 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:25.683521 3903 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:59:25.692229 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:25.693861 3903 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.004s
I20260619 02:59:25.694016 3903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "9e41534198c6455da33c08dd8356cc85"
format_stamp: "Formatted at 2026-06-19 02:59:22 on dist-test-slave-2gms"
server_key: "7afec98b33604566dcf55a38abddf610"
server_key_iv: "d2928fa25d6f2324ff2a6ebc52db4152"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:25.695399 3903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:25.750711 3903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:25.755930 3903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:25.756228 3903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:25.762704 3903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:25.770725 3903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:25.770850 3903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:25.770968 3903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:25.771044 3903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:25.862859 3903 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:41433
I20260619 02:59:25.862881 4063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:41433 every 8 connection(s)
I20260619 02:59:25.864596 3903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:59:25.865272 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 3903
I20260619 02:59:25.866037 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:43385
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:33263
--webserver_port=41819
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:59:25 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837965, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:25.924337 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:25.869906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:40671 (local address 127.22.230.62:33263)
0619 02:59:25.893610 (+ 23704us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:25.893632 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:25.899863 (+ 6231us) server_negotiation.cc:407] Connection header received
0619 02:59:25.902368 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:25.902390 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:25.902936 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:25.903370 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:25.905218 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:25.907205 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:25.909239 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:25.910288 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:25.910474 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:25.914830 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:25.914840 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:25.914857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:25.914920 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:25.918872 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:25.919808 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:25.919817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:25.919826 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:25.920030 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:25.920539 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:25.920544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:25.920548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:25.921164 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:25.921357 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:25.922722 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0619 02:59:25.923143 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23391,"thread_start_us":29193,"threads_started":1}
I20260619 02:59:25.924352 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:25.868216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33263 (local address 127.22.230.1:40671)
0619 02:59:25.898731 (+ 30515us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:25.898800 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:25.901590 (+ 2790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:25.903531 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:25.903544 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:25.904200 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:25.904989 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:25.905015 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:25.907414 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:25.907420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:25.909073 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:25.909086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:25.910885 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:25.910893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:25.911152 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:25.911996 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:25.912027 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:25.914528 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:25.919046 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:25.919055 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:25.919071 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:25.919626 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:25.920172 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:25.920177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:25.920181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:25.920409 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:25.921471 (+ 1062us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:25.921479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:25.921872 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 02:59:25.922196 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 02:59:25.922922 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30192,"thread_start_us":29989,"threads_started":1}
I20260619 02:59:25.927891 4064 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33263
I20260619 02:59:25.928220 4064 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:25.928911 4064 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
W20260619 02:59:25.931883 4064 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:33263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 02:59:25.932116 4064 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:25.932358 4064 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:25.933385 4064 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:25.933629 4064 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
W20260619 02:59:25.934762 4064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 02:59:26.297109 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:26.297456 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:26.297619 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:26.307651 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:26.307833 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:26.308043 4068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:59:26.322718 4068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:43385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=41819
--tserver_master_addrs=127.22.230.62:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:26.324397 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:26.325994 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:26.356685 4084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:26.357431 4085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:26.362833 4068 server_base.cc:1034] running on GCE node
W20260619 02:59:26.363479 4087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:26.364264 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:26.367249 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:26.368538 4068 hybrid_clock.cc:648] HybridClock initialized: now 1781837966368470 us; error 56 us; skew 500 ppm
I20260619 02:59:26.368880 4068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:26 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:26.372768 4068 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:59:26.379125 4068 webserver.cc:460] Webserver started at http://127.22.230.2:41819/ using document root <none> and password file <none>
I20260619 02:59:26.379987 4068 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:26.380127 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:59:26 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:26.413817 4068 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260619 02:59:26.416759 4095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:26.418256 4068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
Jun 19 02:59:26 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781837966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:26.418452 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "8d5ce0ac83454bbbb7e4954587b8498f"
format_stamp: "Formatted at 2026-06-19 02:59:23 on dist-test-slave-2gms"
server_key: "096c9555adec0a437addbc28a90824d4"
server_key_iv: "5f12839dec5627f15681c9348d2aac66"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:26.419660 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:26.458912 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:26.462946 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:26.463244 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:26.464303 4068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:26.465731 4068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:26.465840 4068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:26.466005 4068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:26.466094 4068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:26.521750 4068 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:43385
I20260619 02:59:26.521868 4208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:43385 every 8 connection(s)
I20260619 02:59:26.523460 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:59:26.532282 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4068
I20260619 02:59:26.533056 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.230.20:37481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:45681
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:33263
--webserver_port=41295
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:59:26 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:26.563072 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:26.527332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33263 (local address 127.22.230.2:57253)
0619 02:59:26.533828 (+ 6496us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:26.533882 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:26.536570 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:26.538077 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:26.538090 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:26.538692 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:26.539500 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:26.539534 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:26.541062 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:26.541070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:26.543141 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:26.543158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:26.544395 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:26.544402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:26.544667 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:26.545417 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:26.545438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:26.548309 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:26.557544 (+ 9235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:26.557555 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:26.557574 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:26.558403 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:26.558869 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:26.558876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:26.558883 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:26.559113 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:26.560039 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:26.560051 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:26.560454 (+ 403us) client_negotiation.cc:769] Sending connection context
0619 02:59:26.560897 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 02:59:26.561465 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6185,"thread_start_us":5969,"threads_started":1}
I20260619 02:59:26.564507 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:26.526694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:57253 (local address 127.22.230.62:33263)
0619 02:59:26.537305 (+ 10611us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:26.537314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:26.537337 (+ 23us) server_negotiation.cc:407] Connection header received
0619 02:59:26.537405 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:26.537414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:26.537501 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:26.537611 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:26.539745 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:26.540864 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:26.543316 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:26.544024 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:26.544099 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:26.548578 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:26.548584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:26.548588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:26.548629 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:26.557399 (+ 8770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:26.558515 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:26.558521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:26.558525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:26.558717 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:26.559415 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:26.559422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:26.559427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:26.559820 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:26.559970 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:26.563709 (+ 3739us) server_negotiation.cc:299] Negotiation successful
0619 02:59:26.564013 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10461,"thread_start_us":15403,"threads_started":1}
I20260619 02:59:26.565711 4209 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33263
I20260619 02:59:26.566020 4209 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:26.566970 4209 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:26.569101 3866 ts_manager.cc:194] Registered new tserver with Master: 8d5ce0ac83454bbbb7e4954587b8498f (127.22.230.2:43385)
I20260619 02:59:26.572428 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:57253
Jun 19 02:59:26 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:26.934917 4064 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:26.935369 4064 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:26.936602 3866 ts_manager.cc:194] Registered new tserver with Master: 9e41534198c6455da33c08dd8356cc85 (127.22.230.1:41433)
I20260619 02:59:26.937853 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:40671
W20260619 02:59:26.975037 4213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:26.975369 4213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:26.975522 4213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:26.986135 4213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:26.986251 4213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:26.986413 4213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:59:27.000118 4213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.230.20:37481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:45681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=41295
--tserver_master_addrs=127.22.230.62:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:27.001574 4213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:27.003130 4213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:27.018144 4218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:27.021461 4219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:27.026845 4213 server_base.cc:1034] running on GCE node
W20260619 02:59:27.026691 4221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:27.027732 4213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:27.030823 4213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:27.032091 4213 hybrid_clock.cc:648] HybridClock initialized: now 1781837967032023 us; error 58 us; skew 500 ppm
I20260619 02:59:27.032477 4213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:27 dist-test-slave-2gms krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837967, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:27.039134 4213 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:59:27.040855 4213 webserver.cc:460] Webserver started at http://127.22.230.3:41295/ using document root <none> and password file <none>
I20260619 02:59:27.041667 4213 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:27.041805 4213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 02:59:27 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837967, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:27.067832 4213 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 02:59:27.071803 4229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:27.073432 4213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:59:27.073596 4213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "2fdec5fe2ec84a5396e0a2e204078f9d"
format_stamp: "Formatted at 2026-06-19 02:59:24 on dist-test-slave-2gms"
server_key: "224d3145ba4e6956201a2e2cd93fddaa"
server_key_iv: "9fd86a76ca02d0520a40493c48de8cff"
server_key_version: "kuduclusterkey@0"
I20260619 02:59:27.074479 4213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 02:59:27 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837966, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for HTTP/127.22.230.20@KRBTEST.COM
I20260619 02:59:27.101648 4213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:27.106053 4213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:27.106313 4213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:27.107786 4213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:27.109884 4213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:27.109990 4213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:27.110093 4213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:27.110172 4213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:27.254817 4213 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:45681
I20260619 02:59:27.254985 4342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:45681 every 8 connection(s)
I20260619 02:59:27.256628 4213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:59:27.262342 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4213
Jun 19 02:59:27 dist-test-slave-2gms krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837967, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:27.340368 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.263730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.22.230.62:33263)
0619 02:59:27.308987 (+ 45257us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.308993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.309013 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:59:27.309087 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.309092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.309175 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.309284 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:27.315661 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.316768 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.319633 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.320267 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.320382 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.326225 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:27.326233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:27.326238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:27.326278 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:27.329912 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.334942 (+ 5030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.334947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.334950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.335092 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.336465 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.336475 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.336478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.336925 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:27.339563 (+ 2638us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.339700 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.339985 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45141,"thread_start_us":43305,"threads_started":1}
I20260619 02:59:27.343658 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.260446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:46161 (local address 127.22.230.62:33263)
0619 02:59:27.306802 (+ 46356us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.306808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.314287 (+ 7479us) server_negotiation.cc:407] Connection header received
0619 02:59:27.314355 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.314360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.314424 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.314540 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:27.326919 (+ 12379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.328749 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.331652 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.332287 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.332374 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.336498 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:27.336505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:27.336510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:27.336573 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:27.340084 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.340999 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.341004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.341008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.341141 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.341664 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.341669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.341672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.341992 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:27.342110 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.343080 (+ 970us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.343294 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46211,"thread_start_us":48938,"threads_started":1}
I20260619 02:59:27.345077 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.259939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:33263 (local address 127.22.230.3:46161)
0619 02:59:27.311216 (+ 51277us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.311272 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:27.313873 (+ 2601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:27.321654 (+ 7781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:27.321666 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:27.322329 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:27.326640 (+ 4311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.326667 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.328944 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.328949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.330860 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.330875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.332683 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.332689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.332973 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.333626 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:27.333654 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:27.336232 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:27.340248 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:27.340258 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:27.340289 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:27.340862 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:27.341290 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:27.341295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:27.341299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:27.341533 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:27.342202 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:27.342210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:27.342649 (+ 439us) client_negotiation.cc:769] Sending connection context
0619 02:59:27.343529 (+ 880us) client_negotiation.cc:240] Negotiation successful
0619 02:59:27.344069 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50944,"thread_start_us":58574,"threads_started":1}
I20260619 02:59:27.346459 4343 heartbeater.cc:347] Connected to a master server at 127.22.230.62:33263
I20260619 02:59:27.346820 4343 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:27.347632 4343 heartbeater.cc:511] Master 127.22.230.62:33263 requested a full tablet report, sending...
I20260619 02:59:27.349134 3866 ts_manager.cc:194] Registered new tserver with Master: 2fdec5fe2ec84a5396e0a2e204078f9d (127.22.230.3:45681)
I20260619 02:59:27.350145 3866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:46161
I20260619 02:59:27.355535 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:59:27.372407 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.358732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.22.230.62:33263)
0619 02:59:27.358884 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.358890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.360375 (+ 1485us) server_negotiation.cc:407] Connection header received
0619 02:59:27.360566 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.360572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.360640 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.360734 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:27.362089 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.362855 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.364172 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.364560 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.364710 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.367405 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:27.367410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:27.367413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:27.367441 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:27.369883 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.370509 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.370515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.370518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.370700 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.371054 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.371060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.371063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.371319 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:27.371544 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.371833 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.372036 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 02:59:27.376950 3866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:59:27.385062 3866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:59:27.435495 4367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.417091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:43385 (local address 127.0.0.1:39258)
0619 02:59:27.428413 (+ 11322us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.428432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:27.428550 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:27.429192 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:27.429197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:27.429224 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:27.429563 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.429575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.431252 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.431263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.432841 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.432854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.434465 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.434471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.434669 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.434721 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 02:59:27.434846 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 02:59:27.435038 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11192,"mutex_wait_us":3,"thread_start_us":9318,"threads_started":1}
I20260619 02:59:27.435542 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.417680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.22.230.2:43385)
0619 02:59:27.427016 (+ 9336us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.427028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.428680 (+ 1652us) server_negotiation.cc:407] Connection header received
0619 02:59:27.428778 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.428787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.429001 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.429189 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:27.429732 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.431104 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.432997 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.434253 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.434384 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.434594 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.435067 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.435230 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9148,"thread_start_us":8998,"threads_started":1}
I20260619 02:59:27.436033 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.417108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.22.230.3:45681)
0619 02:59:27.427476 (+ 10368us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.427488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.427535 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:59:27.427826 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.427834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.428103 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.428292 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:27.429081 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.430325 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.433322 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.434620 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.434771 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.434967 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.435597 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.435742 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10143,"thread_start_us":9973,"threads_started":1}
I20260619 02:59:27.436054 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.416683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.22.230.1:41433)
0619 02:59:27.428080 (+ 11397us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.428090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.428120 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:59:27.428216 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.428223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.428458 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.428650 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:27.429309 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.430796 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.433492 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.434342 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.434470 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.434695 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.435593 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.435752 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11182,"thread_start_us":10998,"threads_started":1}
I20260619 02:59:27.436532 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.416864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:45681 (local address 127.0.0.1:56292)
0619 02:59:27.427439 (+ 10575us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.427462 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:27.427603 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:27.428400 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:27.428407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:27.428436 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:27.428911 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.428923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.430461 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.430467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.433149 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.433163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.434863 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.434869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.435130 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.435174 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 02:59:27.435251 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 02:59:27.435371 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9688,"mutex_wait_us":194,"thread_start_us":11286,"threads_started":1}
I20260619 02:59:27.436553 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.416080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:37936)
0619 02:59:27.425899 (+ 9819us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.425945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:27.426128 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:27.428661 (+ 2533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:27.428665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:27.428686 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:27.429096 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.429112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.430950 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.430955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.433343 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.433352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.434895 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.434901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.435145 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.435202 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 02:59:27.435280 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 02:59:27.435402 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9491,"mutex_wait_us":103,"thread_start_us":9352,"threads_started":1}
I20260619 02:59:27.439646 3999 tablet_service.cc:1467] Processing CreateTablet for tablet 71f79dc241394b399281579562a91ac7 (DEFAULT_TABLE table=test-table [id=ae467b289ac24a869df1ee1154f46e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:27.439769 4278 tablet_service.cc:1467] Processing CreateTablet for tablet 71f79dc241394b399281579562a91ac7 (DEFAULT_TABLE table=test-table [id=ae467b289ac24a869df1ee1154f46e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:27.439697 4143 tablet_service.cc:1467] Processing CreateTablet for tablet 71f79dc241394b399281579562a91ac7 (DEFAULT_TABLE table=test-table [id=ae467b289ac24a869df1ee1154f46e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:27.440912 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f79dc241394b399281579562a91ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:27.440930 4278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f79dc241394b399281579562a91ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:27.440930 4143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f79dc241394b399281579562a91ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:27.451287 4371 tablet_bootstrap.cc:492] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Bootstrap starting.
I20260619 02:59:27.454622 4372 tablet_bootstrap.cc:492] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Bootstrap starting.
I20260619 02:59:27.455543 4371 tablet_bootstrap.cc:654] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:27.455674 4373 tablet_bootstrap.cc:492] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Bootstrap starting.
I20260619 02:59:27.456889 4371 log.cc:824] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:27.458058 4372 tablet_bootstrap.cc:654] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:27.458753 4373 tablet_bootstrap.cc:654] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:27.459223 4372 log.cc:824] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:27.459683 4373 log.cc:824] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:27.459967 4371 tablet_bootstrap.cc:492] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: No bootstrap required, opened a new log
I20260619 02:59:27.460263 4371 ts_tablet_manager.cc:1397] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 02:59:27.461789 4372 tablet_bootstrap.cc:492] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: No bootstrap required, opened a new log
I20260619 02:59:27.462036 4372 ts_tablet_manager.cc:1397] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 02:59:27.462563 4373 tablet_bootstrap.cc:492] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: No bootstrap required, opened a new log
I20260619 02:59:27.462858 4373 ts_tablet_manager.cc:1397] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.006s
I20260619 02:59:27.467715 4371 raft_consensus.cc:348] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.468063 4371 raft_consensus.cc:374] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:27.468178 4371 raft_consensus.cc:729] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e41534198c6455da33c08dd8356cc85, State: Initialized, Role: FOLLOWER
I20260619 02:59:27.468750 4371 consensus_queue.cc:260] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.469689 4064 heartbeater.cc:503] Master 127.22.230.62:33263 was elected leader, sending a full tablet report...
I20260619 02:59:27.470260 4371 ts_tablet_manager.cc:1428] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 02:59:27.471916 4372 raft_consensus.cc:348] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.472254 4372 raft_consensus.cc:374] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:27.472409 4372 raft_consensus.cc:729] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5ce0ac83454bbbb7e4954587b8498f, State: Initialized, Role: FOLLOWER
I20260619 02:59:27.472628 4373 raft_consensus.cc:348] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.473006 4373 raft_consensus.cc:374] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:27.472944 4372 consensus_queue.cc:260] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.473130 4373 raft_consensus.cc:729] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdec5fe2ec84a5396e0a2e204078f9d, State: Initialized, Role: FOLLOWER
I20260619 02:59:27.473752 4373 consensus_queue.cc:260] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.474329 4209 heartbeater.cc:503] Master 127.22.230.62:33263 was elected leader, sending a full tablet report...
I20260619 02:59:27.475072 4372 ts_tablet_manager.cc:1428] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260619 02:59:27.475859 4343 heartbeater.cc:503] Master 127.22.230.62:33263 was elected leader, sending a full tablet report...
I20260619 02:59:27.476891 4373 ts_tablet_manager.cc:1428] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260619 02:59:27.537725 4210 tablet.cc:2369] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:59:27.574892 4344 tablet.cc:2369] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:27.615545 4378 raft_consensus.cc:482] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:59:27.615768 4378 raft_consensus.cc:504] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.617417 4378 leader_election.cc:288] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e41534198c6455da33c08dd8356cc85 (127.22.230.1:41433), 2fdec5fe2ec84a5396e0a2e204078f9d (127.22.230.3:45681)
W20260619 02:59:27.623165 4065 tablet.cc:2369] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:27.626243 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.618105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:43621 (local address 127.22.230.3:45681)
0619 02:59:27.618315 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.618322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.618891 (+ 569us) server_negotiation.cc:407] Connection header received
0619 02:59:27.619012 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.619020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.619088 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.619170 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:27.619855 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.620900 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.623904 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.625121 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.625283 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.625361 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.625886 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.626006 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 02:59:27.626245 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.617748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:41061 (local address 127.22.230.1:41433)
0619 02:59:27.617975 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.617985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.618594 (+ 609us) server_negotiation.cc:407] Connection header received
0619 02:59:27.618836 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.618841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.618949 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.619038 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:27.619672 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.620818 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.624057 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.625228 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.625369 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.625444 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.625880 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.626006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 02:59:27.626490 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.617745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:45681 (local address 127.22.230.2:43621)
0619 02:59:27.618797 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.618811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:27.618904 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:27.619203 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:27.619207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:27.619231 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:27.619709 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.619724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.621009 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.621016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.623739 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.623748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.625413 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.625425 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.625653 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.625734 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 02:59:27.625835 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 02:59:27.626024 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":354,"thread_start_us":417,"threads_started":1}
I20260619 02:59:27.626530 4380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.617726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.22.230.2:41061)
0619 02:59:27.618493 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.618507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:27.618692 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:27.619049 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:27.619052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:27.619071 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:27.619528 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.619541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.620948 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.620978 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.623732 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:27.623748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.625441 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.625445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:27.625646 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.625731 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 02:59:27.625847 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 02:59:27.626024 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":275,"thread_start_us":789,"threads_started":1}
I20260619 02:59:27.627812 4298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f79dc241394b399281579562a91ac7" candidate_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" is_pre_election: true
I20260619 02:59:27.627812 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f79dc241394b399281579562a91ac7" candidate_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e41534198c6455da33c08dd8356cc85" is_pre_election: true
I20260619 02:59:27.628353 4298 raft_consensus.cc:2454] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5ce0ac83454bbbb7e4954587b8498f in term 0.
I20260619 02:59:27.628355 4019 raft_consensus.cc:2454] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d5ce0ac83454bbbb7e4954587b8498f in term 0.
I20260619 02:59:27.629019 4099 leader_election.cc:302] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d5ce0ac83454bbbb7e4954587b8498f, 9e41534198c6455da33c08dd8356cc85; no voters:
I20260619 02:59:27.629415 4378 raft_consensus.cc:2781] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:59:27.629566 4378 raft_consensus.cc:482] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:59:27.629666 4378 raft_consensus.cc:3037] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:27.631801 4378 raft_consensus.cc:504] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.632326 4378 leader_election.cc:288] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [CANDIDATE]: Term 1 election: Requested vote from peers 9e41534198c6455da33c08dd8356cc85 (127.22.230.1:41433), 2fdec5fe2ec84a5396e0a2e204078f9d (127.22.230.3:45681)
I20260619 02:59:27.632794 4298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f79dc241394b399281579562a91ac7" candidate_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d"
I20260619 02:59:27.632794 4019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f79dc241394b399281579562a91ac7" candidate_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e41534198c6455da33c08dd8356cc85"
I20260619 02:59:27.632969 4298 raft_consensus.cc:3037] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:27.632983 4019 raft_consensus.cc:3037] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:27.635071 4019 raft_consensus.cc:2454] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5ce0ac83454bbbb7e4954587b8498f in term 1.
I20260619 02:59:27.635074 4298 raft_consensus.cc:2454] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d5ce0ac83454bbbb7e4954587b8498f in term 1.
I20260619 02:59:27.635492 4097 leader_election.cc:302] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fdec5fe2ec84a5396e0a2e204078f9d, 8d5ce0ac83454bbbb7e4954587b8498f; no voters:
I20260619 02:59:27.635792 4378 raft_consensus.cc:2781] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:27.636204 4378 raft_consensus.cc:686] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 1 LEADER]: Becoming Leader. State: Replica: 8d5ce0ac83454bbbb7e4954587b8498f, State: Running, Role: LEADER
I20260619 02:59:27.636637 4378 consensus_queue.cc:237] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } }
I20260619 02:59:27.641466 3865 catalog_manager.cc:5495] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d5ce0ac83454bbbb7e4954587b8498f (127.22.230.2). New cstate: current_term: 1 leader_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5ce0ac83454bbbb7e4954587b8498f" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43385 } health_report { overall_health: HEALTHY } } }
I20260619 02:59:27.658164 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.653126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.22.230.2:43385)
0619 02:59:27.653284 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.653288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.653306 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:59:27.653486 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.653490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.653559 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.653669 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:59:27.653963 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.654739 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.656179 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.656722 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.656872 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.657254 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:59:27.657710 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:59:27.657810 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.657918 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.657975 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 02:59:27.665370 4019 raft_consensus.cc:1261] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 1 FOLLOWER]: Refusing update from remote peer 8d5ce0ac83454bbbb7e4954587b8498f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:27.665372 4298 raft_consensus.cc:1261] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 1 FOLLOWER]: Refusing update from remote peer 8d5ce0ac83454bbbb7e4954587b8498f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:27.666421 4382 consensus_queue.cc:1035] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fdec5fe2ec84a5396e0a2e204078f9d" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:27.666841 4378 consensus_queue.cc:1035] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e41534198c6455da33c08dd8356cc85" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:27.676880 4385 mvcc.cc:205] Tried to move back new op lower bound from 7298408315548381184 to 7298408315443257344. Current Snapshot: MvccSnapshot[applied={T|T < 7298408315548381184}]
I20260619 02:59:27.677438 4386 mvcc.cc:205] Tried to move back new op lower bound from 7298408315548381184 to 7298408315443257344. Current Snapshot: MvccSnapshot[applied={T|T < 7298408315548381184}]
I20260619 02:59:27.683777 4387 mvcc.cc:205] Tried to move back new op lower bound from 7298408315548381184 to 7298408315443257344. Current Snapshot: MvccSnapshot[applied={T|T < 7298408315548381184}]
I20260619 02:59:27.690348 3865 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:59:27.690639 3865 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:59:27.693869 3865 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e2eb497bbb14af6a7cfd63a81f793c9: Sending DeleteTablet for 3 replicas of tablet 71f79dc241394b399281579562a91ac7
I20260619 02:59:27.694796 3999 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f79dc241394b399281579562a91ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:27 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:37936
I20260619 02:59:27.694938 4143 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f79dc241394b399281579562a91ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:27 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:39258
I20260619 02:59:27.695549 4278 tablet_service.cc:1514] Processing DeleteTablet for tablet 71f79dc241394b399281579562a91ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:27 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56292
I20260619 02:59:27.696380 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 3903
I20260619 02:59:27.697355 4397 tablet_replica.cc:330] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: stopping tablet replica
I20260619 02:59:27.697803 4397 raft_consensus.cc:2229] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:59:27.698308 4397 raft_consensus.cc:2258] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:27.699034 4396 tablet_replica.cc:330] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: stopping tablet replica
I20260619 02:59:27.699371 4396 raft_consensus.cc:2229] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:59:27.699911 4396 raft_consensus.cc:2258] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:27.700516 4395 tablet_replica.cc:330] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: stopping tablet replica
I20260619 02:59:27.700912 4395 raft_consensus.cc:2229] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:59:27.701305 4395 raft_consensus.cc:2258] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:27.703635 4396 ts_tablet_manager.cc:1905] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:27.704998 4397 ts_tablet_manager.cc:1905] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:27.711033 4396 ts_tablet_manager.cc:1918] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:27.711658 4397 ts_tablet_manager.cc:1918] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:27.711862 4397 log.cc:1196] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/71f79dc241394b399281579562a91ac7
I20260619 02:59:27.712335 4395 ts_tablet_manager.cc:1905] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:27.712437 4397 ts_tablet_manager.cc:1939] T 71f79dc241394b399281579562a91ac7 P 2fdec5fe2ec84a5396e0a2e204078f9d: Deleting consensus metadata
I20260619 02:59:27.714066 3851 catalog_manager.cc:4841] TS 2fdec5fe2ec84a5396e0a2e204078f9d (127.22.230.3:45681): tablet 71f79dc241394b399281579562a91ac7 (table test-table [id=ae467b289ac24a869df1ee1154f46e1a]) successfully deleted
I20260619 02:59:27.714522 4396 log.cc:1196] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/71f79dc241394b399281579562a91ac7
I20260619 02:59:27.715116 4396 ts_tablet_manager.cc:1939] T 71f79dc241394b399281579562a91ac7 P 8d5ce0ac83454bbbb7e4954587b8498f: Deleting consensus metadata
I20260619 02:59:27.716572 3852 catalog_manager.cc:4841] TS 8d5ce0ac83454bbbb7e4954587b8498f (127.22.230.2:43385): tablet 71f79dc241394b399281579562a91ac7 (table test-table [id=ae467b289ac24a869df1ee1154f46e1a]) successfully deleted
I20260619 02:59:27.719110 4395 ts_tablet_manager.cc:1918] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:27.719321 4395 log.cc:1196] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/71f79dc241394b399281579562a91ac7
I20260619 02:59:27.719547 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.701911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.22.230.1:41433)
0619 02:59:27.702188 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.702194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.702212 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:59:27.702266 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.702270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.702339 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.702449 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:27.705152 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.706221 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.708134 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.708820 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.708935 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.712416 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:27.712428 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:27.712442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:27.712490 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:27.716306 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.717151 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.717160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.717163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.717323 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.717759 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.717765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.717768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.718209 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:27.718498 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.718854 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.719101 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260619 02:59:27.719945 4395 ts_tablet_manager.cc:1939] T 71f79dc241394b399281579562a91ac7 P 9e41534198c6455da33c08dd8356cc85: Deleting consensus metadata
I20260619 02:59:27.720199 4059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:27.809823 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 3903
I20260619 02:59:27.831758 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4068
W20260619 02:59:27.831717 3853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.230.1:41433 (error 108)
I20260619 02:59:27.833798 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.833323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:37964)
0619 02:59:27.833582 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.833663 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260619 02:59:27.834074 3853 catalog_manager.cc:4584] TS 9e41534198c6455da33c08dd8356cc85 (127.22.230.1:41433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71f79dc241394b399281579562a91ac7: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
I20260619 02:59:27.847047 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.832890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.22.230.2:43385)
0619 02:59:27.833085 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.833092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.833112 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:59:27.833176 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.833183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.833272 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.833387 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:27.835147 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.835973 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.837863 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.838300 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.838459 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.841175 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:27.841195 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:27.841210 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:27.841245 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:27.844211 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.844883 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.844894 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.844902 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.845043 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.845427 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.845433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.845437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.845775 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:27.846180 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.846378 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.846671 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 02:59:27.847673 4204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:27.887746 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.887239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:37966)
0619 02:59:27.887460 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.887573 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 02:59:27.888738 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.888372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:37982)
0619 02:59:27.888516 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.888594 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 02:59:27.949019 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.948547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:37988)
0619 02:59:27.948750 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.948859 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 02:59:27.949999 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.949642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:37996)
0619 02:59:27.949794 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:27.949872 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:59:27.954334 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4068
I20260619 02:59:27.972342 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4213
I20260619 02:59:27.984534 4369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:27.973099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.22.230.3:45681)
0619 02:59:27.973246 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:27.973250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:27.973267 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:59:27.973419 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:27.973423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:27.973491 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:27.973588 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:27.974881 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.975554 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.976725 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:27.977168 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:27.977256 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:27.979800 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:27.979811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:27.979825 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:27.979863 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:27.982300 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.982962 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.982969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.982973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.983099 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:27.983392 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:27.983395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:27.983397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:27.983710 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:27.983918 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:27.984118 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 02:59:27.984253 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 02:59:27.985098 4338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:28.030485 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:28.029953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:38006)
0619 02:59:28.030202 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:28.030301 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 02:59:28.031477 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:28.031083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41433 (local address 127.0.0.1:38016)
0619 02:59:28.031285 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:28.031360 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 02:59:28.077306 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4213
I20260619 02:59:28.095547 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 3833
I20260619 02:59:28.096650 3896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:28.201784 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 3833
I20260619 02:59:28.221930 23448 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 02:59:28.557638 23448 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 02:59:28.557821 23448 mini_ranger.cc:66] Stopping Ranger...
I20260619 02:59:28.902746 23448 mini_ranger.cc:68] Stopped Ranger
2026-06-19 02:59:28.903 UTC [2885] LOG: received smart shutdown request
2026-06-19 02:59:28.906 UTC [2885] LOG: background worker "logical replication launcher" (PID 2894) exited with exit code 1
2026-06-19 02:59:28.906 UTC [2889] LOG: shutting down
2026-06-19 02:59:29.048 UTC [2885] LOG: database system is shut down
2026-06-19T02:59:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50343 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:59:29 dist-test-slave-2gms krb5kdc[4411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:59:29 dist-test-slave-2gms krb5kdc[4411](info): set up 2 sockets
Jun 19 02:59:29 dist-test-slave-2gms krb5kdc[4411](info): commencing operation
krb5kdc: starting...
W20260619 02:59:30.218819 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:59:30 dist-test-slave-2gms krb5kdc[4411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:59:30.424062 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:41135
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:41135,127.22.230.61:35461,127.22.230.60:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:30.723330 4424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:30.723551 4424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:30.723626 4424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:30.730129 4424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:30.730208 4424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:30.730257 4424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:30.730309 4424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:30.730345 4424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:30.739470 4424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:41135,127.22.230.61:35461,127.22.230.60:39683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:30.740619 4424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:30.741869 4424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:30.752796 4429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:30.752749 4432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:30.752835 4430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:30.753464 4424 server_base.cc:1034] running on GCE node
I20260619 02:59:30.754127 4424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:30.755409 4424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:30.756630 4424 hybrid_clock.cc:648] HybridClock initialized: now 1781837970756598 us; error 45 us; skew 500 ppm
I20260619 02:59:30.756956 4424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:30 dist-test-slave-2gms krb5kdc[4411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:30.759891 4424 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:30.761338 4424 webserver.cc:460] Webserver started at http://127.22.230.62:41735/ using document root <none> and password file <none>
I20260619 02:59:30.761924 4424 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:30.762015 4424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:30.762235 4424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:30.764894 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "2b756eff72c1420b949aa7518f258295"
format_stamp: "Formatted at 2026-06-19 02:59:30 on dist-test-slave-2gms"
server_key: "527facad49b292dc158d150c23e2914d"
server_key_iv: "01b25b13dce131753e9074389e22b64e"
server_key_version: "encryptionkey@0"
I20260619 02:59:30.765618 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "2b756eff72c1420b949aa7518f258295"
format_stamp: "Formatted at 2026-06-19 02:59:30 on dist-test-slave-2gms"
server_key: "527facad49b292dc158d150c23e2914d"
server_key_iv: "01b25b13dce131753e9074389e22b64e"
server_key_version: "encryptionkey@0"
I20260619 02:59:30.770329 4424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260619 02:59:30.773546 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:30.774777 4424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:30.774943 4424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "2b756eff72c1420b949aa7518f258295"
format_stamp: "Formatted at 2026-06-19 02:59:30 on dist-test-slave-2gms"
server_key: "527facad49b292dc158d150c23e2914d"
server_key_iv: "01b25b13dce131753e9074389e22b64e"
server_key_version: "encryptionkey@0"
I20260619 02:59:30.775094 4424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:30.790395 4424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:30.793888 4424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:30.794152 4424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:30.811839 4424 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:41135
I20260619 02:59:30.811882 4490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:41135 every 8 connection(s)
I20260619 02:59:30.813051 4424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:30.816183 4491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:30.821151 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4424
I20260619 02:59:30.821388 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:59:30.821995 23448 external_mini_cluster.cc:1428] Setting key 785586876398b8f63fa73f2609c8bb67
I20260619 02:59:30.822199 4491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:59:30 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.230.61@KRBTEST.COM; defaulting to no policy
I20260619 02:59:30.843686 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.826280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.22.230.62:41135)
0619 02:59:30.827012 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:30.827020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:30.827067 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:59:30.828078 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:30.828084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:30.828289 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:30.828480 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:30.829592 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:30.831367 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:30.832637 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:30.833138 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:30.833337 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:30.836933 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:30.836941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:30.836950 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:30.836977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:30.839778 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:30.840693 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:30.840698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:30.840700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:30.840828 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:30.841259 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:30.841262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:30.841263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:30.841666 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:30.841822 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:30.842453 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 02:59:30.842761 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
I20260619 02:59:30.843732 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.825057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:41135 (local address 127.0.0.1:41828)
0619 02:59:30.825746 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.825787 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:30.827748 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:30.828480 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:30.828488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:30.829120 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:30.829410 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:30.829426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:30.831510 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:30.831515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:30.832505 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:30.832515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:30.833370 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:30.833377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:30.833553 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:30.834157 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:30.834178 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:30.836719 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:30.839922 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:30.839937 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:30.839950 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:30.840573 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:30.840944 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:30.840949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:30.840954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:30.841154 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:30.841829 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:30.841851 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:30.842089 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 02:59:30.842353 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 02:59:30.842947 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":99,"thread_start_us":299,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.61@KRBTEST.COM" created.
I20260619 02:59:30.846449 4491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:59:30.847610 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.847144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38388)
0619 02:59:30.847331 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.847458 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 02:59:30.847892 4440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
I20260619 02:59:30.849472 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.849012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38400)
0619 02:59:30.849201 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.849290 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:59:30.849696 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:35461: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab.
Entry for principal kudu/127.22.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab.
I20260619 02:59:30.869470 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:35461
--webserver_interface=127.22.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab
--principal=kudu/127.22.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:41135,127.22.230.61:35461,127.22.230.60:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:59:30.899025 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } attempt: 1
I20260619 02:59:30.900077 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.899606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38410)
0619 02:59:30.899797 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.899930 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:59:30.901007 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.900638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38422)
0619 02:59:30.900784 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.900865 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:59:30.901197 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:35461: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
I20260619 02:59:30.969446 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } attempt: 2
I20260619 02:59:30.970727 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.970169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38432)
0619 02:59:30.970415 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.970529 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 02:59:30.971691 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:30.971300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38442)
0619 02:59:30.971474 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:30.971552 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 02:59:30.971895 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:35461: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
I20260619 02:59:31.063154 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } attempt: 3
I20260619 02:59:31.064476 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.063949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38444)
0619 02:59:31.064182 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.064300 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 02:59:31.065459 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.065086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38456)
0619 02:59:31.065229 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.065306 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:59:31.065645 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:35461: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
W20260619 02:59:31.176357 4498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:31.176596 4498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:31.176677 4498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:31.183521 4498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:31.183611 4498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:31.183661 4498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:31.183714 4498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:31.183758 4498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:31.192680 4498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:41135,127.22.230.61:35461,127.22.230.60:39683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab
--principal=kudu/127.22.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:35461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:31.193758 4498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:31.194866 4498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:31.205711 4503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:31.205719 4504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:31.205834 4506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:31.206281 4498 server_base.cc:1034] running on GCE node
I20260619 02:59:31.207089 4498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:31.208336 4498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:31.208876 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } attempt: 4
I20260619 02:59:31.209583 4498 hybrid_clock.cc:648] HybridClock initialized: now 1781837971209541 us; error 45 us; skew 500 ppm
I20260619 02:59:31.209960 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.209407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38468)
0619 02:59:31.209697 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.209795 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260619 02:59:31.209947 4498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 02:59:31.210971 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.210665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38472)
0619 02:59:31.210787 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.210865 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 02:59:31.211155 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:35461: Network error: Client connection negotiation failed: client connection to 127.22.230.61:35461: connect: Connection refused (error 111)
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:31.213263 4498 init.cc:377] Logged in from keytab as kudu/127.22.230.61@KRBTEST.COM (short username kudu)
I20260619 02:59:31.214635 4498 webserver.cc:460] Webserver started at http://127.22.230.61:33789/ using document root <none> and password file <none>
I20260619 02:59:31.215210 4498 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:31.215335 4498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:31.215600 4498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:31.218156 4498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data/instance:
uuid: "a86759edf39447ddbd9441ff1330c718"
format_stamp: "Formatted at 2026-06-19 02:59:31 on dist-test-slave-2gms"
server_key: "c8aa13608cd4a6390e323294ad16e57c"
server_key_iv: "fb5228808c0d4247eb2faa14df550c19"
server_key_version: "encryptionkey@0"
I20260619 02:59:31.218925 4498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal/instance:
uuid: "a86759edf39447ddbd9441ff1330c718"
format_stamp: "Formatted at 2026-06-19 02:59:31 on dist-test-slave-2gms"
server_key: "c8aa13608cd4a6390e323294ad16e57c"
server_key_iv: "fb5228808c0d4247eb2faa14df550c19"
server_key_version: "encryptionkey@0"
I20260619 02:59:31.223295 4498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 02:59:31.226087 4513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:31.227325 4498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 02:59:31.227500 4498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal
uuid: "a86759edf39447ddbd9441ff1330c718"
format_stamp: "Formatted at 2026-06-19 02:59:31 on dist-test-slave-2gms"
server_key: "c8aa13608cd4a6390e323294ad16e57c"
server_key_iv: "fb5228808c0d4247eb2faa14df550c19"
server_key_version: "encryptionkey@0"
I20260619 02:59:31.227676 4498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:31.239181 4498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:31.243012 4498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:31.243245 4498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:31.260512 4498 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.61:35461
I20260619 02:59:31.260541 4564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.61:35461 every 8 connection(s)
I20260619 02:59:31.261713 4498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
I20260619 02:59:31.262902 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4498
I20260619 02:59:31.263085 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-1/wal/instance
I20260619 02:59:31.263584 23448 external_mini_cluster.cc:1428] Setting key e280394aa6fe8c13241818be873ccf56
I20260619 02:59:31.265689 4565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:31.274315 4565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.60@KRBTEST.COM" created.
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:31.296878 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.276697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.22.230.62:41135)
0619 02:59:31.276933 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.276940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.278079 (+ 1139us) server_negotiation.cc:407] Connection header received
0619 02:59:31.280862 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.280867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.280935 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.281017 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.282973 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.283563 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.285870 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.286266 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.286359 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.290968 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.290973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.290976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.291008 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.293574 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.294458 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.294468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.294470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.294644 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.295062 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.295065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.295067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.295330 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.295431 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.296338 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.296563 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 02:59:31.297724 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.277106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:41135 (local address 127.0.0.1:41836)
0619 02:59:31.277913 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.277960 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.280606 (+ 2646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.281285 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.281297 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.282045 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.282801 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.282828 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.283714 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.283721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.285658 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.285673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.286493 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.286501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.286843 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.287506 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.287527 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.290759 (+ 3232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.293712 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.293723 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.293738 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.294350 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.294738 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.294745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.294750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.294965 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.295455 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.295465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.295888 (+ 423us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.296204 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.296629 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":289,"threads_started":1}
I20260619 02:59:31.299469 4565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 02:59:31.314297 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.300196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38488)
0619 02:59:31.300354 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.300374 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.300491 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.301316 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.301319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.301684 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.301811 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.301818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.303959 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.303962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.304731 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.304740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.306005 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.306013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.306192 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.306801 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.306816 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.308900 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.312224 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.312227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.312229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.312572 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.312912 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.312915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.312917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.313042 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.313545 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.313548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.313645 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.313747 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.313974 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:59:31.314643 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.300260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.22.230.61:35461)
0619 02:59:31.300849 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.300858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.300888 (+ 30us) server_negotiation.cc:407] Connection header received
0619 02:59:31.300939 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.300950 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.301156 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.301312 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.301972 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.303832 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.304861 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.305813 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.305947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.309116 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.309132 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.309148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.309187 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.312093 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.312682 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.312689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.312691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.312834 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.313134 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.313137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.313138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.313436 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.313586 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.313892 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.314110 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab.
Entry for principal kudu/127.22.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab.
I20260619 02:59:31.317032 4565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:59:31.317799 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:39683
--webserver_interface=127.22.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab
--principal=kudu/127.22.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:41135,127.22.230.61:35461,127.22.230.60:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:59:31.318082 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.317636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52864)
0619 02:59:31.317792 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.317894 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 02:59:31.318317 4517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.319794 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.319487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52880)
0619 02:59:31.319610 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.319708 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:59:31.319996 4565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.369289 4565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 1
I20260619 02:59:31.370451 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.369922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52892)
0619 02:59:31.370142 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.370263 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:59:31.371454 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.371053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52904)
0619 02:59:31.371224 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.371320 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 02:59:31.371668 4565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.439904 4565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 2
I20260619 02:59:31.441084 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.440565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52918)
0619 02:59:31.440773 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.440891 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 02:59:31.442054 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.441688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52922)
0619 02:59:31.441844 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.441927 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:59:31.442265 4565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.510638 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } attempt: 5
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 02:59:31.525059 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.511385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38490)
0619 02:59:31.511616 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.511638 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.511754 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.512082 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.512085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.512419 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.512599 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.512606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.513973 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.513977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.514947 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.514958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.516051 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.516055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.516176 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.516792 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.516808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.519137 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.522846 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.522849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.522852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.523189 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.523576 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.523579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.523582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.523694 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.524330 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.524333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.524444 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.524620 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.524823 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 02:59:31.525247 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.511566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.22.230.61:35461)
0619 02:59:31.511754 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.511760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.511784 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:59:31.511846 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.511852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.511945 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.512051 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.512758 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.513836 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.515082 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.515908 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.516064 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.519330 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.519338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.519345 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.519398 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.522714 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.523301 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.523307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.523322 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.523468 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.523800 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.523807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.523812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.524217 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.524384 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.524743 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.524948 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 02:59:31.526319 4491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:59:31.527382 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.527048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52936)
0619 02:59:31.527193 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.527261 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 02:59:31.528203 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.527913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52944)
0619 02:59:31.528044 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.528121 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:59:31.528410 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.533463 4565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 3
I20260619 02:59:31.534409 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.533910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52956)
0619 02:59:31.534098 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.534208 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 02:59:31.535480 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.534999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52962)
0619 02:59:31.535228 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.535321 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260619 02:59:31.535672 4565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.579614 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 1
I20260619 02:59:31.580716 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.580181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52978)
0619 02:59:31.580411 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.580525 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 02:59:31.581775 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.581354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52980)
0619 02:59:31.581524 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.581627 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 02:59:31.582020 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
W20260619 02:59:31.630676 4572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:31.631011 4572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:31.631148 4572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:31.637943 4572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:31.638077 4572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:31.638185 4572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:31.638293 4572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:31.638362 4572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:31.647423 4572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:41135,127.22.230.61:35461,127.22.230.60:39683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab
--principal=kudu/127.22.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:31.648593 4572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:31.649765 4572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 02:59:31.650207 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 2
I20260619 02:59:31.651412 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.650854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52986)
0619 02:59:31.651093 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.651197 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 02:59:31.652474 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.652068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:52988)
0619 02:59:31.652223 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.652322 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 02:59:31.652690 4491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
W20260619 02:59:31.661440 4577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:31.661479 4578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:31.661406 4580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:31.662173 4572 server_base.cc:1034] running on GCE node
I20260619 02:59:31.662820 4572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:31.663932 4572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:31.665135 4572 hybrid_clock.cc:648] HybridClock initialized: now 1781837971665077 us; error 40 us; skew 500 ppm
I20260619 02:59:31.665473 4572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:31.668387 4572 init.cc:377] Logged in from keytab as kudu/127.22.230.60@KRBTEST.COM (short username kudu)
I20260619 02:59:31.669649 4572 webserver.cc:460] Webserver started at http://127.22.230.60:32831/ using document root <none> and password file <none>
I20260619 02:59:31.670233 4572 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:31.670320 4572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:31.670553 4572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:31.673369 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data/instance:
uuid: "633d5e760a174861b31b18f4dfcb26b5"
format_stamp: "Formatted at 2026-06-19 02:59:31 on dist-test-slave-2gms"
server_key: "a47562a825b308b6cbf8287aa968aaee"
server_key_iv: "31e379aab174d90fc26465b2445ed9c2"
server_key_version: "encryptionkey@0"
I20260619 02:59:31.674108 4572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal/instance:
uuid: "633d5e760a174861b31b18f4dfcb26b5"
format_stamp: "Formatted at 2026-06-19 02:59:31 on dist-test-slave-2gms"
server_key: "a47562a825b308b6cbf8287aa968aaee"
server_key_iv: "31e379aab174d90fc26465b2445ed9c2"
server_key_version: "encryptionkey@0"
I20260619 02:59:31.678886 4572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 02:59:31.678864 4565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 4
I20260619 02:59:31.679764 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.679400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:53004)
0619 02:59:31.679573 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.679642 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 02:59:31.680588 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.680338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:53008)
0619 02:59:31.680451 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.680509 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 02:59:31.680811 4565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:39683: Network error: Client connection negotiation failed: client connection to 127.22.230.60:39683: connect: Connection refused (error 111)
I20260619 02:59:31.682365 4587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:31.683538 4572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 02:59:31.683703 4572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal
uuid: "633d5e760a174861b31b18f4dfcb26b5"
format_stamp: "Formatted at 2026-06-19 02:59:31 on dist-test-slave-2gms"
server_key: "a47562a825b308b6cbf8287aa968aaee"
server_key_iv: "31e379aab174d90fc26465b2445ed9c2"
server_key_version: "encryptionkey@0"
I20260619 02:59:31.683883 4572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:31.707268 4572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:31.710811 4572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:31.711074 4572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:31.728483 4572 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.60:39683
I20260619 02:59:31.728554 4638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.60:39683 every 8 connection(s)
I20260619 02:59:31.729640 4572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
I20260619 02:59:31.732861 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:31.738238 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4572
I20260619 02:59:31.738447 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781837774831470-23448-0/minicluster-data/master-2/wal/instance
I20260619 02:59:31.738998 23448 external_mini_cluster.cc:1428] Setting key 8e5f48820f99229ce1d20250834280c4
I20260619 02:59:31.738873 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 02:59:31.757176 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.741637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.22.230.62:41135)
0619 02:59:31.741784 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.741788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.742437 (+ 649us) server_negotiation.cc:407] Connection header received
0619 02:59:31.743754 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.743760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.743839 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.743935 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.745070 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.745825 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.747372 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.747801 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.747936 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.751674 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.751681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.751686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.751722 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.754268 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.755069 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.755074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.755078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.755199 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.755595 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.755599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.755602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.755883 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.756049 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.756621 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.756847 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 02:59:31.758177 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.741669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:41135 (local address 127.0.0.1:41850)
0619 02:59:31.742284 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.742313 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.743530 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.743996 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.744002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.744547 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.744908 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.744923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.745985 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.745996 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.747234 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.747245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.748019 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.748028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.748249 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.748995 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.749014 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.751485 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.754390 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.754399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.754408 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.754931 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.755313 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.755318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.755322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.755484 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.756011 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.756017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.756262 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.756527 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.756887 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
I20260619 02:59:31.758610 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.740733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.22.230.60:39683)
0619 02:59:31.741551 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.741568 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.741614 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:59:31.743520 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.743539 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.743749 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.743941 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.745457 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.746626 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.748000 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.748504 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.748626 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.753039 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.753046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.753054 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.753079 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.755729 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.756418 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.756425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.756428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.756582 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.756959 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.756963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.756966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.757370 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.757650 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.757941 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.758175 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260619 02:59:31.758912 4491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 3
I20260619 02:59:31.759675 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:31.773221 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.759556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.22.230.60:39683)
0619 02:59:31.759706 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.759713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.759730 (+ 17us) server_negotiation.cc:407] Connection header received
0619 02:59:31.759768 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.759772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.759832 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.759921 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.760753 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.762018 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.763155 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.763834 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.763993 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.767219 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.767226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.767230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.767266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.770428 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.771248 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.771253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.771258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.771403 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.771804 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.771809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.771813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.772070 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.772214 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.772681 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.772869 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260619 02:59:31.773257 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.759333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:53034)
0619 02:59:31.759572 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.759590 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.759677 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.759953 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.759958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.760395 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.760604 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.760617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.762128 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.762132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.763024 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.763034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.764012 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.764018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.764183 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.764739 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.764759 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.767026 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.770555 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.770560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.770589 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.771121 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.771541 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.771548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.771555 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.771703 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.772189 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.772194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.772396 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.772551 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.772796 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260619 02:59:31.774976 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.760212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:35461 (local address 127.0.0.1:38506)
0619 02:59:31.760425 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.760442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.760522 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.760849 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.760854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.761298 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.761450 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.761460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.762716 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.762721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.763798 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.763809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.765740 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.765749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.765953 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.766452 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.766483 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.768706 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.772291 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.772297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.772301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.772749 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.773421 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.773424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.773426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.773560 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.774112 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.774117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.774270 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.774398 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.774684 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":3}
I20260619 02:59:31.775275 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.760283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.22.230.61:35461)
0619 02:59:31.760480 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.760487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.760507 (+ 20us) server_negotiation.cc:407] Connection header received
0619 02:59:31.760626 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.760630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.760703 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.760827 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.761573 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.762519 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.763927 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.764544 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.764665 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.768897 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.768902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.768905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.768933 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.772169 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.772872 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.772876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.772878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.773061 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.773660 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.773665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.773667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.773980 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.774126 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.774726 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.774911 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 02:59:31.777829 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.761386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.22.230.62:41135)
0619 02:59:31.761541 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.761545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.761561 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:59:31.761695 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.761700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.761754 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.761832 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.763295 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.764289 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.765618 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.765982 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.766108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.770998 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.771005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.771011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.771075 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.775569 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.776138 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.776143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.776147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.776292 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.776594 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.776598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.776601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.776939 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.777208 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.777350 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.777532 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:59:31.778890 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } has no permanent_uuid. Determining permanent_uuid...
I20260619 02:59:31.779529 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: Bootstrap starting.
I20260619 02:59:31.783179 4491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:31.784446 4491 log.cc:824] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 02:59:31.789063 4491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: No bootstrap required, opened a new log
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 02:59:31.793491 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.779486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:53048)
0619 02:59:31.779612 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.779627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.779716 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.780166 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.780170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.780532 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.780691 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.780702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.781983 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.781988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.783060 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.783070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.783734 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.783739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.783960 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.785055 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.785070 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.787348 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.790240 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.790247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.790251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.790760 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.791424 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.791431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.791436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.791585 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.792456 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.792462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.792611 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.792779 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.793064 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260619 02:59:31.793573 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.779840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.22.230.60:39683)
0619 02:59:31.779970 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.779973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.779984 (+ 11us) server_negotiation.cc:407] Connection header received
0619 02:59:31.780025 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.780027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.780065 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.780155 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.780832 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.781829 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.783182 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.783610 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.783727 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.787559 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.787566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.787571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.787610 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.790121 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.790878 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.790884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.790887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.791298 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.791706 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.791711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.791715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.792320 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.792471 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.792893 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.793230 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":28}
I20260619 02:59:31.796072 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.779797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.22.230.61:35461)
0619 02:59:31.779947 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.779952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.780299 (+ 347us) server_negotiation.cc:407] Connection header received
0619 02:59:31.780345 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.780349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.780421 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.780508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.781640 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.782530 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.784355 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.784812 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.784894 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.790362 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.790369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.790382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.790419 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.793426 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.794199 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.794202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.794204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.794310 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.794744 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.794749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.794753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.795079 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.795322 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.795577 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.795776 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:59:31.798532 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: Bootstrap starting.
I20260619 02:59:31.799018 4491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:31.799356 4491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:31.799489 4491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b756eff72c1420b949aa7518f258295, State: Initialized, Role: FOLLOWER
I20260619 02:59:31.800069 4491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:31.800765 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:59:31.800989 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:31.801347 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:31.802003 4491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:31.802457 4639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:31.803679 4639 log.cc:824] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:31.809528 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: No bootstrap required, opened a new log
I20260619 02:59:31.823537 4639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:31.824734 4639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:31.824906 4639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633d5e760a174861b31b18f4dfcb26b5, State: Initialized, Role: FOLLOWER
I20260619 02:59:31.825634 4639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:31.837612 4662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:31.838110 4662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:31.840893 4639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:31.846084 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.810618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.22.230.62:41135)
0619 02:59:31.810777 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.810782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.824509 (+ 13727us) server_negotiation.cc:407] Connection header received
0619 02:59:31.824686 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.824691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.824769 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.824870 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.827207 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.828518 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.831353 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.832262 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.832380 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.837526 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.837535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.837543 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.837596 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.841313 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.842210 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.842215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.842219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.842437 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.843095 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.843100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.843102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.843451 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.843719 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.845494 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.845719 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 02:59:31.853188 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.827192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.22.230.60:39683)
0619 02:59:31.827343 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.827349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.837567 (+ 10218us) server_negotiation.cc:407] Connection header received
0619 02:59:31.837983 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.837988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.838078 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.838159 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.840015 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.841161 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.843098 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.843667 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.843764 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.846693 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.846700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.846707 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.846743 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.850121 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.850919 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.850924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.850928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.851065 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.851473 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.851478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.851481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.851832 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.852070 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.852603 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.852831 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 02:59:31.858368 4663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:59:31.858469 4663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:59:31.846849 4424 master_runner.cc:186] Error getting master registration for 127.22.230.61:35461: OK, instance_id { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" instance_seqno: 1781837971255667 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:59:31.860743 4424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:59:31.862272 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.827599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.22.230.61:35461)
0619 02:59:31.827750 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.827755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.834331 (+ 6576us) server_negotiation.cc:407] Connection header received
0619 02:59:31.834523 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.834533 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.834652 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.834777 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.847425 (+ 12648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.848572 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.850401 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.851060 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.851148 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.855735 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.855744 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.855761 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.855806 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.859153 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.860028 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.860034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.860037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.860187 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.860594 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.860599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.860602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.860999 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.861203 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.861718 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.861955 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 02:59:31.874310 4677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 02:59:31.874449 4677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:59:31.870886 4572 master_runner.cc:186] Error getting master registration for 127.22.230.61:35461: OK, instance_id { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" instance_seqno: 1781837971255667 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 02:59:31.882229 4572 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 02:59:31.963038 4593 tablet.cc:2369] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:31.980019 4565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } attempt: 5
Jun 19 02:59:31 dist-test-slave-2gms krb5kdc[4411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837971, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 02:59:31.995858 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.980529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:39683 (local address 127.0.0.1:53074)
0619 02:59:31.980772 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:31.980791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:31.980891 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:31.981239 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:31.981244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:31.981759 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:31.981958 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.981986 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.983435 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.983441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.984881 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:31.984895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.985857 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.985865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:31.986054 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.986692 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:31.986713 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:31.989494 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:31.993242 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.993247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.993250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.993712 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.994093 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:31.994098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:31.994101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:31.994240 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:31.994918 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:31.994924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:31.995054 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 02:59:31.995201 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 02:59:31.995474 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 02:59:31.996722 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:31.980614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.22.230.60:39683)
0619 02:59:31.980790 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:31.980797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:31.980843 (+ 46us) server_negotiation.cc:407] Connection header received
0619 02:59:31.981013 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:31.981018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:31.981094 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:31.981197 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:31.982129 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.983271 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.985049 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:31.985714 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:31.985837 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:31.989681 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:31.989687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:31.989692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:31.989725 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:31.993103 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.993835 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.993840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.993845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.993971 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:31.994344 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:31.994348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:31.994351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:31.994726 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:31.994864 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:31.996177 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0619 02:59:31.996391 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 02:59:32.002300 4565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: Bootstrap starting.
I20260619 02:59:32.006225 4565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:32.007545 4565 log.cc:824] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:32.010514 4565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: No bootstrap required, opened a new log
I20260619 02:59:32.020789 4565 raft_consensus.cc:348] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:32.021153 4565 raft_consensus.cc:374] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:32.021261 4565 raft_consensus.cc:729] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86759edf39447ddbd9441ff1330c718, State: Initialized, Role: FOLLOWER
I20260619 02:59:32.021852 4565 consensus_queue.cc:260] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:32.022779 4680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.023222 4680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:32.023955 4565 sys_catalog.cc:564] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:32.037137 4498 master_runner.cc:263] No leader master found from master a86759edf39447ddbd9441ff1330c718
I20260619 02:59:32.037263 4498 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 02:59:32.038228 4662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:59:32.038537 4662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:32.039935 4662 leader_election.cc:288] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b756eff72c1420b949aa7518f258295 (127.22.230.62:41135), a86759edf39447ddbd9441ff1330c718 (127.22.230.61:35461)
I20260619 02:59:32.040776 4466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "633d5e760a174861b31b18f4dfcb26b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b756eff72c1420b949aa7518f258295" is_pre_election: true
I20260619 02:59:32.041303 4466 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633d5e760a174861b31b18f4dfcb26b5 in term 0.
I20260619 02:59:32.041910 4591 leader_election.cc:302] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b756eff72c1420b949aa7518f258295, 633d5e760a174861b31b18f4dfcb26b5; no voters:
W20260619 02:59:32.043047 4691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 02:59:32.042897 4540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "633d5e760a174861b31b18f4dfcb26b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a86759edf39447ddbd9441ff1330c718" is_pre_election: true
W20260619 02:59:32.043174 4691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 02:59:32.043387 4540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633d5e760a174861b31b18f4dfcb26b5 in term 0.
I20260619 02:59:32.044194 4662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:59:32.044337 4662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:59:32.044422 4662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 0 FOLLOWER]: Advancing to term 1
W20260619 02:59:32.046618 4445 tablet.cc:2369] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:32.047430 4662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:32.048431 4466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "633d5e760a174861b31b18f4dfcb26b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b756eff72c1420b949aa7518f258295"
I20260619 02:59:32.048615 4466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:32.053439 4466 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633d5e760a174861b31b18f4dfcb26b5 in term 1.
I20260619 02:59:32.055331 4591 leader_election.cc:302] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b756eff72c1420b949aa7518f258295, 633d5e760a174861b31b18f4dfcb26b5; no voters:
I20260619 02:59:32.057695 4540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "633d5e760a174861b31b18f4dfcb26b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a86759edf39447ddbd9441ff1330c718"
I20260619 02:59:32.057883 4540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:32.061556 4693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:32.068527 4540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633d5e760a174861b31b18f4dfcb26b5 in term 1.
I20260619 02:59:32.069051 4662 leader_election.cc:288] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2b756eff72c1420b949aa7518f258295 (127.22.230.62:41135), a86759edf39447ddbd9441ff1330c718 (127.22.230.61:35461)
I20260619 02:59:32.073886 4693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [term 1 LEADER]: Becoming Leader. State: Replica: 633d5e760a174861b31b18f4dfcb26b5, State: Running, Role: LEADER
I20260619 02:59:32.074417 4693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } }
I20260619 02:59:32.078402 4662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 633d5e760a174861b31b18f4dfcb26b5. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.078697 4662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:32.087141 4696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:32.089344 4696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:32.104166 4540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [term 1 FOLLOWER]: Refusing update from remote peer 633d5e760a174861b31b18f4dfcb26b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:32.105286 4466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [term 1 FOLLOWER]: Refusing update from remote peer 633d5e760a174861b31b18f4dfcb26b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:32.106316 4662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:32.107702 4662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:32.154450 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 633d5e760a174861b31b18f4dfcb26b5. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.154811 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:32.158131 4697 mvcc.cc:205] Tried to move back new op lower bound from 7298408333726838784 to 7298408333620883456. Current Snapshot: MvccSnapshot[applied={T|T < 7298408333726838784}]
I20260619 02:59:32.164469 4699 mvcc.cc:205] Tried to move back new op lower bound from 7298408333726838784 to 7298408333620883456. Current Snapshot: MvccSnapshot[applied={T|T < 7298408333726838784}]
I20260619 02:59:32.169791 4680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 633d5e760a174861b31b18f4dfcb26b5. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.170125 4680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:32.171937 4694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.172365 4694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:32.173334 4696 catalog_manager.cc:1347] Generated new cluster ID: 063aa65c9c494e7b80f0a258be8ab922
I20260619 02:59:32.173441 4696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:59:32.173609 4694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.173840 4694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:32.176859 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.177080 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:32.179145 4698 mvcc.cc:205] Tried to move back new op lower bound from 7298408333726838784 to 7298408333620883456. Current Snapshot: MvccSnapshot[applied={T|T < 7298408333726838784}]
I20260619 02:59:32.185632 4680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "633d5e760a174861b31b18f4dfcb26b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b756eff72c1420b949aa7518f258295" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 41135 } } peers { permanent_uuid: "a86759edf39447ddbd9441ff1330c718" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 35461 } } peers { permanent_uuid: "633d5e760a174861b31b18f4dfcb26b5" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 39683 } } }
I20260619 02:59:32.185838 4680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 02:59:32.214812 4696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:59:32.216671 4696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:59:32.232838 4696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5: Generated new TSK 0
I20260619 02:59:32.234669 4696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:59:32.860219 4663 catalog_manager.cc:1259] Loaded cluster ID: 063aa65c9c494e7b80f0a258be8ab922
I20260619 02:59:32.860304 4663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: loading cluster ID for follower catalog manager: success
I20260619 02:59:32.862728 4663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: acquiring CA information for follower catalog manager: success
I20260619 02:59:32.864346 4663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b756eff72c1420b949aa7518f258295: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:59:33.044850 4691 catalog_manager.cc:1259] Loaded cluster ID: 063aa65c9c494e7b80f0a258be8ab922
I20260619 02:59:33.044950 4691 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: loading cluster ID for follower catalog manager: success
I20260619 02:59:33.047353 4691 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: acquiring CA information for follower catalog manager: success
I20260619 02:59:33.049149 4691 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a86759edf39447ddbd9441ff1330c718: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 02:59:33.366140 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4424
I20260619 02:59:33.367518 4486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:33.472287 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4424
I20260619 02:59:33.492828 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4498
I20260619 02:59:33.493871 4560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:33.564970 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:33.564202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:41135 (local address 127.0.0.1:41862)
0619 02:59:33.564713 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:33.564823 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":296,"threads_started":1}
W20260619 02:59:33.565269 4591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.62:41135: connect: Connection refused (error 111)
I20260619 02:59:33.566498 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:33.566186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:41135 (local address 127.0.0.1:41878)
0619 02:59:33.566315 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:33.566387 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 02:59:33.566691 4591 consensus_peers.cc:487] T 00000000000000000000000000000000 P 633d5e760a174861b31b18f4dfcb26b5 -> Peer 2b756eff72c1420b949aa7518f258295 (127.22.230.62:41135): Couldn't send request to peer 2b756eff72c1420b949aa7518f258295. Status: Network error: Client connection negotiation failed: client connection to 127.22.230.62:41135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 02:59:33.595431 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4498
I20260619 02:59:33.616065 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4572
I20260619 02:59:33.617033 4634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:33.721745 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4572
2026-06-19T02:59:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4553 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:59:33 dist-test-slave-2gms krb5kdc[4715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:59:33 dist-test-slave-2gms krb5kdc[4715](info): set up 2 sockets
Jun 19 02:59:33 dist-test-slave-2gms krb5kdc[4715](info): commencing operation
krb5kdc: starting...
W20260619 02:59:34.783020 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:59:34 dist-test-slave-2gms krb5kdc[4715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:59:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:59:34.993224 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33175
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:33175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:35.300761 4728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:35.301120 4728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:35.301270 4728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:35.307909 4728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:35.308040 4728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:35.308146 4728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:35.308248 4728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:35.308334 4728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:35.317312 4728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:33175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:35.318492 4728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:35.319757 4728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:35.331179 4736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:35.331393 4733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:35.331357 4734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:35.332541 4728 server_base.cc:1034] running on GCE node
I20260619 02:59:35.333200 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:35.334337 4728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:35.335525 4728 hybrid_clock.cc:648] HybridClock initialized: now 1781837975335456 us; error 59 us; skew 500 ppm
I20260619 02:59:35.335850 4728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:35 dist-test-slave-2gms krb5kdc[4715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837975, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:35.338774 4728 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:35.340204 4728 webserver.cc:460] Webserver started at http://127.22.230.62:45595/ using document root <none> and password file <none>
I20260619 02:59:35.340837 4728 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:35.340968 4728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:35.341257 4728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:35.344055 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "d2fe61a769ef40e39c384bcea239fef5"
format_stamp: "Formatted at 2026-06-19 02:59:35 on dist-test-slave-2gms"
server_key: "44e167c47a14dfd1d92ff4f74589a03c"
server_key_iv: "960e8a6105cd58a797144ef7eda670d6"
server_key_version: "encryptionkey@0"
I20260619 02:59:35.344789 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "d2fe61a769ef40e39c384bcea239fef5"
format_stamp: "Formatted at 2026-06-19 02:59:35 on dist-test-slave-2gms"
server_key: "44e167c47a14dfd1d92ff4f74589a03c"
server_key_iv: "960e8a6105cd58a797144ef7eda670d6"
server_key_version: "encryptionkey@0"
I20260619 02:59:35.349519 4728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 02:59:35.352605 4743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:35.353787 4728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 02:59:35.353960 4728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "d2fe61a769ef40e39c384bcea239fef5"
format_stamp: "Formatted at 2026-06-19 02:59:35 on dist-test-slave-2gms"
server_key: "44e167c47a14dfd1d92ff4f74589a03c"
server_key_iv: "960e8a6105cd58a797144ef7eda670d6"
server_key_version: "encryptionkey@0"
I20260619 02:59:35.354159 4728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:35.364581 4728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:35.368024 4728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:35.368291 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:35.386113 4728 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:33175
I20260619 02:59:35.386142 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:33175 every 8 connection(s)
I20260619 02:59:35.387441 4728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:35.390012 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4728
I20260619 02:59:35.390244 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:59:35.390959 4795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:35.390966 23448 external_mini_cluster.cc:1428] Setting key 6ecb4dee503ef5fbf305dedd6fa38a16
I20260619 02:59:35.404158 4795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5: Bootstrap starting.
Jun 19 02:59:35 dist-test-slave-2gms krb5kdc[4715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:35.407672 4795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:35.408969 4795 log.cc:824] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:35.412137 4795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5: No bootstrap required, opened a new log
I20260619 02:59:35.413715 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:35.392584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.22.230.62:33175)
0619 02:59:35.393268 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:35.393278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:35.393315 (+ 37us) server_negotiation.cc:407] Connection header received
0619 02:59:35.395099 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:35.395116 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:35.395433 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:35.395705 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:35.397390 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:35.398686 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:35.400507 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:35.401152 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:35.401298 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:35.405602 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:35.405613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:35.405633 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:35.405677 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:35.409488 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:35.410339 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:35.410350 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:35.410360 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:35.410546 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:35.410906 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:35.410910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:35.410913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:35.411516 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:35.411796 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:35.412073 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 02:59:35.412470 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":329,"threads_started":1}
I20260619 02:59:35.417358 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 02:59:35.420670 4795 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fe61a769ef40e39c384bcea239fef5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33175 } }
I20260619 02:59:35.421022 4795 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:35.421149 4795 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fe61a769ef40e39c384bcea239fef5, State: Initialized, Role: FOLLOWER
I20260619 02:59:35.421663 4795 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fe61a769ef40e39c384bcea239fef5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33175 } }
I20260619 02:59:35.421900 4795 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:59:35.422037 4795 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:59:35.422185 4795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:35.424221 4795 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fe61a769ef40e39c384bcea239fef5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33175 } }
I20260619 02:59:35.424669 4795 leader_election.cc:302] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2fe61a769ef40e39c384bcea239fef5; no voters:
I20260619 02:59:35.425213 4795 leader_election.cc:288] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:59:35.425346 4800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:35.425994 4800 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [term 1 LEADER]: Becoming Leader. State: Replica: d2fe61a769ef40e39c384bcea239fef5, State: Running, Role: LEADER
I20260619 02:59:35.426466 4800 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fe61a769ef40e39c384bcea239fef5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33175 } }
I20260619 02:59:35.427146 4795 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:35.429056 4801 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2fe61a769ef40e39c384bcea239fef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fe61a769ef40e39c384bcea239fef5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33175 } } }
I20260619 02:59:35.429436 4801 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:35.429358 4802 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2fe61a769ef40e39c384bcea239fef5. Latest consensus state: current_term: 1 leader_uuid: "d2fe61a769ef40e39c384bcea239fef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2fe61a769ef40e39c384bcea239fef5" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33175 } } }
I20260619 02:59:35.429594 4802 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:35.433313 4808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:35.437716 4808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:35.444859 4808 catalog_manager.cc:1347] Generated new cluster ID: 945eeb481086463e9af0150bd5599ef3
I20260619 02:59:35.444944 4808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:59:35.482470 4808 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:59:35.483722 4808 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:59:35.493752 4808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2fe61a769ef40e39c384bcea239fef5: Generated new TSK 0
I20260619 02:59:35.494472 4808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 02:59:35.528787 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4728
I20260619 02:59:35.529739 4790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:35.619871 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4728
2026-06-19T02:59:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1894 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:59:35 dist-test-slave-2gms krb5kdc[4823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:59:35 dist-test-slave-2gms krb5kdc[4823](info): set up 2 sockets
Jun 19 02:59:35 dist-test-slave-2gms krb5kdc[4823](info): commencing operation
krb5kdc: starting...
W20260619 02:59:36.691098 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 02:59:36 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T02:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T02:59:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 02:59:36.897219 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45261
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:37.202440 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:37.202713 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:37.202869 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:37.209395 4836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:37.209482 4836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:37.209525 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:37.209602 4836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:37.209708 4836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:37.218328 4836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:37.219475 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:37.220700 4836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:37.231885 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:37.231854 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:37.231817 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:37.232458 4836 server_base.cc:1034] running on GCE node
I20260619 02:59:37.233117 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:37.234201 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:37.235440 4836 hybrid_clock.cc:648] HybridClock initialized: now 1781837977235388 us; error 37 us; skew 500 ppm
I20260619 02:59:37.235749 4836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:37 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837977, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:37.238693 4836 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:37.239933 4836 webserver.cc:460] Webserver started at http://127.22.230.62:46681/ using document root <none> and password file <none>
I20260619 02:59:37.240551 4836 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:37.240694 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:37.240981 4836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:37.243578 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "9ed0f13cf5fd4324a4f716caebdd10a6"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "2eb00ee538491dbfcab446092bc081b8"
server_key_iv: "ad60ebc7b6598bb743a0553e870e7793"
server_key_version: "encryptionkey@0"
I20260619 02:59:37.244282 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "9ed0f13cf5fd4324a4f716caebdd10a6"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "2eb00ee538491dbfcab446092bc081b8"
server_key_iv: "ad60ebc7b6598bb743a0553e870e7793"
server_key_version: "encryptionkey@0"
I20260619 02:59:37.248817 4836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 02:59:37.251888 4851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:37.253014 4836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 02:59:37.253170 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "9ed0f13cf5fd4324a4f716caebdd10a6"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "2eb00ee538491dbfcab446092bc081b8"
server_key_iv: "ad60ebc7b6598bb743a0553e870e7793"
server_key_version: "encryptionkey@0"
I20260619 02:59:37.253336 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:37.269795 4836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:37.273523 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:37.273762 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:37.290983 4836 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:45261
I20260619 02:59:37.291023 4902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:45261 every 8 connection(s)
I20260619 02:59:37.292127 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:37.295183 4903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:37.297309 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4836
I20260619 02:59:37.297554 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 02:59:37.298352 23448 external_mini_cluster.cc:1428] Setting key 049a24cf12633795e09e6c2301eaab92
I20260619 02:59:37.306186 4903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap starting.
I20260619 02:59:37.308965 4903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:37.309847 4903 log.cc:824] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Log is configured to *not* fsync() on all Append() calls
Jun 19 02:59:37 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:37.312728 4903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: No bootstrap required, opened a new log
I20260619 02:59:37.319891 4903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:37.320236 4903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:37.320358 4903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed0f13cf5fd4324a4f716caebdd10a6, State: Initialized, Role: FOLLOWER
I20260619 02:59:37.320412 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:37.300332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.22.230.62:45261)
0619 02:59:37.301247 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:37.301260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:37.301323 (+ 63us) server_negotiation.cc:407] Connection header received
0619 02:59:37.303203 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:37.303221 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:37.303547 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:37.303932 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:37.305293 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:37.306611 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:37.307972 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:37.308617 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:37.308814 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:37.313473 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:37.313479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:37.313491 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:37.313524 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:37.316517 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:37.317207 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:37.317216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:37.317221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:37.317391 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:37.317786 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:37.317793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:37.317801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:37.318259 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:37.318626 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:37.319028 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 02:59:37.319461 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":395,"threads_started":1}
I20260619 02:59:37.320981 4903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:37.321235 4903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:59:37.321362 4903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:59:37.321519 4903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:37.324046 4903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:37.324436 4903 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed0f13cf5fd4324a4f716caebdd10a6; no voters:
I20260619 02:59:37.324849 4903 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 02:59:37.324997 4908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:37.326232 4908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed0f13cf5fd4324a4f716caebdd10a6, State: Running, Role: LEADER
I20260619 02:59:37.326685 4903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:37.326822 4908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:37.329787 4910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed0f13cf5fd4324a4f716caebdd10a6. Latest consensus state: current_term: 1 leader_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } } }
I20260619 02:59:37.330123 4910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:37.330669 4909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } } }
I20260619 02:59:37.331008 4909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:37.331880 4916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:37.336098 4916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:37.343966 4916 catalog_manager.cc:1347] Generated new cluster ID: b512be3628f94816aa8736aed9f376c9
I20260619 02:59:37.344053 4916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:59:37.354622 4916 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 02:59:37.355731 4916 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:59:37.366231 4916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Generated new TSK 0
I20260619 02:59:37.366982 4916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 02:59:37.437233 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:45261
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:37.774200 4929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:37.774441 4929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:37.774528 4929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:37.781453 4929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:37.781540 4929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:37.781641 4929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:59:37.790838 4929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:37.791875 4929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:37.793006 4929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:37.805167 4935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:37.805212 4934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:37.805126 4937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:37.810880 4929 server_base.cc:1034] running on GCE node
I20260619 02:59:37.811578 4929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:37.812413 4929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:37.813643 4929 hybrid_clock.cc:648] HybridClock initialized: now 1781837977813613 us; error 41 us; skew 500 ppm
I20260619 02:59:37.813977 4929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:37 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837977, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:37.817036 4929 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:59:37.818377 4929 webserver.cc:460] Webserver started at http://127.22.230.1:45069/ using document root <none> and password file <none>
I20260619 02:59:37.819106 4929 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:37.819242 4929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:37.819607 4929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:37.822438 4929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "14b60302e0fa4031a81b6787ece778f4"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "6e9bfbe568f00f2f99a301e665889dd0"
server_key_iv: "bd72a15441d6922dae488f5ad9955ca2"
server_key_version: "encryptionkey@0"
I20260619 02:59:37.823180 4929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "14b60302e0fa4031a81b6787ece778f4"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "6e9bfbe568f00f2f99a301e665889dd0"
server_key_iv: "bd72a15441d6922dae488f5ad9955ca2"
server_key_version: "encryptionkey@0"
I20260619 02:59:37.827775 4929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 02:59:37.830977 4944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:37.832381 4929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 02:59:37.832499 4929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "14b60302e0fa4031a81b6787ece778f4"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "6e9bfbe568f00f2f99a301e665889dd0"
server_key_iv: "bd72a15441d6922dae488f5ad9955ca2"
server_key_version: "encryptionkey@0"
I20260619 02:59:37.832661 4929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:37.854607 4929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:37.858263 4929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:37.858480 4929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:37.859340 4929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:37.860467 4929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:37.860534 4929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:37.860610 4929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:37.860651 4929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:37.891645 4929 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40557
I20260619 02:59:37.891724 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40557 every 8 connection(s)
I20260619 02:59:37.892846 4929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:59:37.901180 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 4929
I20260619 02:59:37.901392 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 02:59:37.902007 23448 external_mini_cluster.cc:1428] Setting key 44b1d1cf42da2505b3892bcc4fa2b7fa
Jun 19 02:59:37 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837977, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:37.911643 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:37.895141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:47793 (local address 127.22.230.62:45261)
0619 02:59:37.895560 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:37.895566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:37.896537 (+ 971us) server_negotiation.cc:407] Connection header received
0619 02:59:37.898228 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:37.898232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:37.898298 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:37.898393 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:37.899826 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:37.900572 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:37.902313 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:37.902827 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:37.902932 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:37.906128 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:37.906132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:37.906136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:37.906166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:37.908672 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:37.909402 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:37.909406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:37.909409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:37.909521 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:37.909943 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:37.909949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:37.909952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:37.910231 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:37.910390 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:37.911076 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 02:59:37.911286 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":199,"threads_started":1}
I20260619 02:59:37.912469 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:37.895687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:47793)
0619 02:59:37.896383 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:37.896429 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:37.897996 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:37.898636 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:37.898646 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:37.899150 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:37.899679 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:37.899694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:37.900798 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:37.900802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:37.902172 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:37.902183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:37.903071 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:37.903078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:37.903276 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:37.903953 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:37.903973 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:37.905926 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:37.908821 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:37.908830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:37.908844 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:37.909263 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:37.909644 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:37.909649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:37.909653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:37.909826 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:37.910360 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:37.910368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:37.910725 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 02:59:37.910982 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 02:59:37.911450 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":267,"threads_started":1}
I20260619 02:59:37.913951 5057 heartbeater.cc:347] Connected to a master server at 127.22.230.62:45261
I20260619 02:59:37.914239 5057 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:37.914964 5057 heartbeater.cc:511] Master 127.22.230.62:45261 requested a full tablet report, sending...
I20260619 02:59:37.917047 4868 ts_manager.cc:194] Registered new tserver with Master: 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557)
I20260619 02:59:37.919276 4868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:47793
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 02:59:37.951799 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:45261
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:38.261706 5063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:38.261937 5063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:38.262023 5063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:38.268613 5063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:38.268715 5063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:38.268810 5063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:59:38.277082 5063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:38.278124 5063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:38.279425 5063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:38.291702 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:38.291693 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:38.292559 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:38.292598 5063 server_base.cc:1034] running on GCE node
I20260619 02:59:38.293169 5063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:38.293998 5063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:38.295240 5063 hybrid_clock.cc:648] HybridClock initialized: now 1781837978295157 us; error 38 us; skew 500 ppm
I20260619 02:59:38.295684 5063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:38 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:38.298679 5063 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:59:38.300112 5063 webserver.cc:460] Webserver started at http://127.22.230.2:42873/ using document root <none> and password file <none>
I20260619 02:59:38.300738 5063 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:38.300865 5063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:38.301187 5063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:38.303865 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "f3c29b2b61374bc9be2752eae75d5d64"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "6a0e8acb567fb9cd8c367ed4f3471ce0"
server_key_iv: "12328619c1f02282eb709ad60d4a4765"
server_key_version: "encryptionkey@0"
I20260619 02:59:38.304563 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c29b2b61374bc9be2752eae75d5d64"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "6a0e8acb567fb9cd8c367ed4f3471ce0"
server_key_iv: "12328619c1f02282eb709ad60d4a4765"
server_key_version: "encryptionkey@0"
I20260619 02:59:38.309274 5063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 02:59:38.312565 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:38.313692 5063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 02:59:38.313864 5063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "f3c29b2b61374bc9be2752eae75d5d64"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "6a0e8acb567fb9cd8c367ed4f3471ce0"
server_key_iv: "12328619c1f02282eb709ad60d4a4765"
server_key_version: "encryptionkey@0"
I20260619 02:59:38.314051 5063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:38.343356 5063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:38.347031 5063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:38.347302 5063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:38.348191 5063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:38.349452 5063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:38.349524 5063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:38.349689 5063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:38.349746 5063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:38.381553 5063 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:44697
I20260619 02:59:38.381601 5190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:44697 every 8 connection(s)
I20260619 02:59:38.382823 5063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:59:38.390974 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5063
I20260619 02:59:38.391152 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 02:59:38.391708 23448 external_mini_cluster.cc:1428] Setting key 4024a0e17c5593e7a61c54fed96d36ca
Jun 19 02:59:38 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:38.401961 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.385120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:52625 (local address 127.22.230.62:45261)
0619 02:59:38.385282 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:38.385286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:38.386373 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 02:59:38.388183 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:38.388191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:38.388288 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:38.388389 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:38.389752 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.390427 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.392577 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.393009 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.393150 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.396401 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:38.396406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:38.396410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:38.396441 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:38.399051 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:38.399732 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:38.399742 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:38.399744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:38.399864 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:38.400298 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:38.400302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:38.400304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:38.400554 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:38.400705 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:38.401349 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 02:59:38.401622 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 02:59:38.402931 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.385604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:52625)
0619 02:59:38.386215 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:38.386258 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:38.387880 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:38.388593 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:38.388600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:38.389080 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:38.389610 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.389626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.390546 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.390549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.392433 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.392457 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.393230 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.393237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.393469 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.394082 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:38.394102 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:38.396185 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:38.399198 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:38.399206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:38.399221 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:38.399608 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:38.399969 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:38.399975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:38.399978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:38.400159 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:38.400677 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:38.400684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:38.401001 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 02:59:38.401244 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 02:59:38.401762 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":251,"threads_started":1}
I20260619 02:59:38.404249 5191 heartbeater.cc:347] Connected to a master server at 127.22.230.62:45261
I20260619 02:59:38.404549 5191 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:38.405242 5191 heartbeater.cc:511] Master 127.22.230.62:45261 requested a full tablet report, sending...
I20260619 02:59:38.406661 4868 ts_manager.cc:194] Registered new tserver with Master: f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697)
I20260619 02:59:38.407472 4868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:52625
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 02:59:38.439759 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:45261
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:38.739650 5196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:38.739873 5196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:38.739956 5196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:38.746582 5196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:38.746681 5196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:38.746800 5196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:59:38.755285 5196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:38.756407 5196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:38.757568 5196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:38.769527 5201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:38.769531 5202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:38.769598 5204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:38.770272 5196 server_base.cc:1034] running on GCE node
I20260619 02:59:38.770912 5196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:38.771766 5196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:38.772994 5196 hybrid_clock.cc:648] HybridClock initialized: now 1781837978772952 us; error 56 us; skew 500 ppm
I20260619 02:59:38.773293 5196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:38 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:38.776726 5196 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:59:38.778088 5196 webserver.cc:460] Webserver started at http://127.22.230.3:43441/ using document root <none> and password file <none>
I20260619 02:59:38.778740 5196 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:38.778841 5196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:38.779076 5196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 02:59:38.781664 5196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "1170175a71534cac872361156a13e56e"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "2546c1c4bed65d5e1b6b7e42ded07d4c"
server_key_iv: "cbe1831dc86e348286468a4b2771b5b8"
server_key_version: "encryptionkey@0"
I20260619 02:59:38.782335 5196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "1170175a71534cac872361156a13e56e"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "2546c1c4bed65d5e1b6b7e42ded07d4c"
server_key_iv: "cbe1831dc86e348286468a4b2771b5b8"
server_key_version: "encryptionkey@0"
I20260619 02:59:38.786970 5196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 02:59:38.790081 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:38.791334 5196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:38.791456 5196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "1170175a71534cac872361156a13e56e"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "2546c1c4bed65d5e1b6b7e42ded07d4c"
server_key_iv: "cbe1831dc86e348286468a4b2771b5b8"
server_key_version: "encryptionkey@0"
I20260619 02:59:38.791627 5196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:38.809644 5196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:38.813572 5196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:38.813802 5196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:38.814615 5196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:38.815776 5196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:38.815843 5196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:38.815919 5196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:38.815969 5196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:38.847409 5196 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:44543
I20260619 02:59:38.847488 5323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:44543 every 8 connection(s)
I20260619 02:59:38.848551 5196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:59:38.857563 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5196
I20260619 02:59:38.857748 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 02:59:38.858314 23448 external_mini_cluster.cc:1428] Setting key 0f6cebee94fc777431415468f4fa5766
Jun 19 02:59:38 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:38.867225 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.851034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:36391 (local address 127.22.230.62:45261)
0619 02:59:38.851202 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:38.851206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:38.852347 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 02:59:38.853995 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:38.853999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:38.854065 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:38.854150 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:38.855559 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.856461 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.858050 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.858704 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.858892 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.862084 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:38.862090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:38.862095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:38.862137 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:38.864393 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:38.865054 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:38.865060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:38.865064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:38.865194 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:38.865601 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:38.865606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:38.865609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:38.865870 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:38.866020 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:38.866671 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 02:59:38.866893 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:59:38.868003 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.851514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:36391)
0619 02:59:38.852208 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:38.852242 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:38.853746 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:38.854329 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:38.854335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:38.854913 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:38.855406 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.855423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.856619 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.856623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.857913 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.857924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.859009 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.859017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.859249 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.859842 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:38.859865 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:38.861896 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:38.864526 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:38.864534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:38.864547 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:38.864929 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:38.865305 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:38.865310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:38.865313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:38.865490 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:38.866006 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:38.866013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:38.866298 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 02:59:38.866538 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 02:59:38.867032 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":265,"threads_started":1}
I20260619 02:59:38.869153 5324 heartbeater.cc:347] Connected to a master server at 127.22.230.62:45261
I20260619 02:59:38.869428 5324 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:38.870074 5324 heartbeater.cc:511] Master 127.22.230.62:45261 requested a full tablet report, sending...
I20260619 02:59:38.871382 4868 ts_manager.cc:194] Registered new tserver with Master: 1170175a71534cac872361156a13e56e (127.22.230.3:44543)
I20260619 02:59:38.872459 4868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:36391
I20260619 02:59:38.873353 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:59:38.888638 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.877526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.22.230.62:45261)
0619 02:59:38.877681 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:38.877685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:38.878165 (+ 480us) server_negotiation.cc:407] Connection header received
0619 02:59:38.878289 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:38.878292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:38.878341 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:38.878414 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:38.879749 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.880411 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.881625 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.881995 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.882077 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.884152 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:38.884157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:38.884160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:38.884185 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:38.886487 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:38.887191 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:38.887195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:38.887197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:38.887307 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:38.887607 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:38.887610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:38.887612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:38.887875 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:38.888044 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:38.888288 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 02:59:38.888422 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:59:38.892364 4867 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:59:38.899289 4867 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:59:38.922068 5057 heartbeater.cc:503] Master 127.22.230.62:45261 was elected leader, sending a full tablet report...
I20260619 02:59:38.931236 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.910547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.22.230.2:44697)
0619 02:59:38.922520 (+ 11973us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:38.922531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:38.923206 (+ 675us) server_negotiation.cc:407] Connection header received
0619 02:59:38.923337 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:38.923348 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:38.923643 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:38.924472 (+ 829us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:38.924713 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.926211 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.928810 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.930125 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.930271 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.930455 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:38.930786 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 02:59:38.930947 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11780,"thread_start_us":12003,"threads_started":1}
I20260619 02:59:38.931254 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.910292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:44697 (local address 127.0.0.1:51946)
0619 02:59:38.922884 (+ 12592us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:38.922907 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:38.923051 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:38.923868 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:38.923874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:38.923902 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:38.924271 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.924283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.926446 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.926451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.928666 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.928677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.930332 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.930338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.930535 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.930610 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 02:59:38.930739 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 02:59:38.930908 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12430,"thread_start_us":11345,"threads_started":1}
I20260619 02:59:38.931967 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.909930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51178)
0619 02:59:38.921207 (+ 11277us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:38.921259 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:38.921449 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:38.924641 (+ 3192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:38.924652 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:38.924679 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:38.925154 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.925166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.927458 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.927466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.929496 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.929510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.931263 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.931270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.931425 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.931464 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 02:59:38.931529 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 02:59:38.931642 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10980,"thread_start_us":12605,"threads_started":1}
I20260619 02:59:38.932819 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.911943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:44543 (local address 127.0.0.1:56684)
0619 02:59:38.924907 (+ 12964us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:38.924923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:38.925044 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:38.925825 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:38.925830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:38.925852 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:38.926251 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.926263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.928271 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.928276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.930291 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:38.930308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.932067 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.932073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:38.932233 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.932262 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 02:59:38.932349 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 02:59:38.932475 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12797,"thread_start_us":12641,"threads_started":1}
I20260619 02:59:38.932997 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.910058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.22.230.1:40557)
0619 02:59:38.924048 (+ 13990us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:38.924059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:38.924087 (+ 28us) server_negotiation.cc:407] Connection header received
0619 02:59:38.924172 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:38.924179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:38.924401 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:38.925326 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:38.926018 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.927325 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.929660 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.931035 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.932281 (+ 1246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.932506 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:38.932562 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 02:59:38.932703 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13787,"thread_start_us":13606,"threads_started":1}
I20260619 02:59:38.933032 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:38.912545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.22.230.3:44543)
0619 02:59:38.924439 (+ 11894us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:38.924447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:38.925159 (+ 712us) server_negotiation.cc:407] Connection header received
0619 02:59:38.925247 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:38.925253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:38.925439 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:38.925593 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:38.926454 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.928126 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.930464 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:38.931870 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:38.932022 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:38.932237 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:38.932496 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 02:59:38.932704 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11725,"thread_start_us":11559,"threads_started":1}
I20260619 02:59:38.935170 5126 tablet_service.cc:1467] Processing CreateTablet for tablet 23ec852ca89e4eb0b2425ec455dbc8d7 (DEFAULT_TABLE table=test-table [id=03c380bb1eda49c5b08182c345c6f83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:38.935849 4992 tablet_service.cc:1467] Processing CreateTablet for tablet 23ec852ca89e4eb0b2425ec455dbc8d7 (DEFAULT_TABLE table=test-table [id=03c380bb1eda49c5b08182c345c6f83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:38.936601 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ec852ca89e4eb0b2425ec455dbc8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:38.937069 4992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ec852ca89e4eb0b2425ec455dbc8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:38.936760 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 23ec852ca89e4eb0b2425ec455dbc8d7 (DEFAULT_TABLE table=test-table [id=03c380bb1eda49c5b08182c345c6f83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:38.938059 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ec852ca89e4eb0b2425ec455dbc8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:38.947005 5342 tablet_bootstrap.cc:492] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Bootstrap starting.
I20260619 02:59:38.950160 5342 tablet_bootstrap.cc:654] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:38.950917 5343 tablet_bootstrap.cc:492] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Bootstrap starting.
I20260619 02:59:38.951416 5342 log.cc:824] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:38.952479 5344 tablet_bootstrap.cc:492] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Bootstrap starting.
I20260619 02:59:38.954938 5343 tablet_bootstrap.cc:654] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:38.955212 5344 tablet_bootstrap.cc:654] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:38.956115 5343 log.cc:824] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:38.956463 5344 log.cc:824] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:38.957546 5342 tablet_bootstrap.cc:492] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: No bootstrap required, opened a new log
I20260619 02:59:38.957862 5342 ts_tablet_manager.cc:1397] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260619 02:59:38.959215 5344 tablet_bootstrap.cc:492] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: No bootstrap required, opened a new log
I20260619 02:59:38.959481 5344 ts_tablet_manager.cc:1397] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 02:59:38.960873 5343 tablet_bootstrap.cc:492] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: No bootstrap required, opened a new log
I20260619 02:59:38.961099 5343 ts_tablet_manager.cc:1397] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260619 02:59:38.966694 5344 raft_consensus.cc:348] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:38.967041 5344 raft_consensus.cc:374] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:38.967162 5344 raft_consensus.cc:729] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1170175a71534cac872361156a13e56e, State: Initialized, Role: FOLLOWER
I20260619 02:59:38.967725 5344 consensus_queue.cc:260] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:38.968758 5324 heartbeater.cc:503] Master 127.22.230.62:45261 was elected leader, sending a full tablet report...
I20260619 02:59:38.968617 5342 raft_consensus.cc:348] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:38.968999 5342 raft_consensus.cc:374] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:38.969103 5342 raft_consensus.cc:729] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b60302e0fa4031a81b6787ece778f4, State: Initialized, Role: FOLLOWER
I20260619 02:59:38.969342 5344 ts_tablet_manager.cc:1428] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 02:59:38.969627 5342 consensus_queue.cc:260] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:38.971417 5342 ts_tablet_manager.cc:1428] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260619 02:59:38.972502 5343 raft_consensus.cc:348] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:38.972877 5343 raft_consensus.cc:374] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:38.973023 5343 raft_consensus.cc:729] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c29b2b61374bc9be2752eae75d5d64, State: Initialized, Role: FOLLOWER
I20260619 02:59:38.973578 5343 consensus_queue.cc:260] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:38.974673 5191 heartbeater.cc:503] Master 127.22.230.62:45261 was elected leader, sending a full tablet report...
I20260619 02:59:38.975415 5343 ts_tablet_manager.cc:1428] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260619 02:59:39.093276 5348 raft_consensus.cc:482] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:59:39.093561 5348 raft_consensus.cc:504] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:39.095239 5348 leader_election.cc:288] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557), f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697)
W20260619 02:59:39.100114 5325 tablet.cc:2369] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:39.102191 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.095490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:44697 (local address 127.22.230.3:35557)
0619 02:59:39.095833 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:39.095870 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:39.096043 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:39.096473 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:39.096480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:39.096551 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:39.096728 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:39.096762 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.098094 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.098098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:39.100156 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:39.100170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.101445 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.101451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:39.101665 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.101706 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 02:59:39.101805 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 02:59:39.101941 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"spinlock_wait_cycles":20864}
I20260619 02:59:39.102231 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.095641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:35557 (local address 127.22.230.2:44697)
0619 02:59:39.095858 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:39.095864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:39.096183 (+ 319us) server_negotiation.cc:407] Connection header received
0619 02:59:39.096264 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:39.096271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:39.096341 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:39.096410 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:39.096916 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.097970 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.100353 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.101293 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.101406 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.101463 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:39.101839 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 02:59:39.101976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 02:59:39.102658 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.095579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.22.230.3:47475)
0619 02:59:39.096341 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:39.096354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:39.096501 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:39.096803 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:39.096810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:39.096841 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:39.097106 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:39.097114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.098371 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.098376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:39.100418 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:39.100430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.101814 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.101819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:39.101993 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.102024 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:59:39.102118 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 02:59:39.102236 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":1,"thread_start_us":457,"threads_started":1}
I20260619 02:59:39.102967 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.095915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:47475 (local address 127.22.230.1:40557)
0619 02:59:39.096132 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:39.096137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:39.096483 (+ 346us) server_negotiation.cc:407] Connection header received
0619 02:59:39.096632 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:39.096636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:39.096690 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:39.096757 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:39.097295 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.098248 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.100574 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.101604 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.101762 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.101841 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:39.102159 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 02:59:39.102726 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 02:59:39.103233 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ec852ca89e4eb0b2425ec455dbc8d7" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c29b2b61374bc9be2752eae75d5d64" is_pre_election: true
I20260619 02:59:39.103713 5146 raft_consensus.cc:2454] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 0.
I20260619 02:59:39.103931 5012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ec852ca89e4eb0b2425ec455dbc8d7" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b60302e0fa4031a81b6787ece778f4" is_pre_election: true
I20260619 02:59:39.104421 5213 leader_election.cc:302] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1170175a71534cac872361156a13e56e, f3c29b2b61374bc9be2752eae75d5d64; no voters:
I20260619 02:59:39.104465 5012 raft_consensus.cc:2454] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 0.
I20260619 02:59:39.104784 5348 raft_consensus.cc:2781] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:59:39.104943 5348 raft_consensus.cc:482] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:59:39.105098 5348 raft_consensus.cc:3037] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:39.107290 5348 raft_consensus.cc:504] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:39.107826 5348 leader_election.cc:288] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 election: Requested vote from peers 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557), f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697)
I20260619 02:59:39.108234 5012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ec852ca89e4eb0b2425ec455dbc8d7" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b60302e0fa4031a81b6787ece778f4"
I20260619 02:59:39.108259 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ec852ca89e4eb0b2425ec455dbc8d7" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c29b2b61374bc9be2752eae75d5d64"
I20260619 02:59:39.108429 5012 raft_consensus.cc:3037] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:39.108426 5146 raft_consensus.cc:3037] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:39.110455 5012 raft_consensus.cc:2454] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 1.
I20260619 02:59:39.110499 5146 raft_consensus.cc:2454] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 1.
I20260619 02:59:39.110901 5212 leader_election.cc:302] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1170175a71534cac872361156a13e56e, 14b60302e0fa4031a81b6787ece778f4; no voters:
I20260619 02:59:39.111191 5348 raft_consensus.cc:2781] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:39.111618 5348 raft_consensus.cc:686] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 1 LEADER]: Becoming Leader. State: Replica: 1170175a71534cac872361156a13e56e, State: Running, Role: LEADER
I20260619 02:59:39.112033 5348 consensus_queue.cc:237] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:39.116314 4867 catalog_manager.cc:5495] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1170175a71534cac872361156a13e56e (127.22.230.3). New cstate: current_term: 1 leader_uuid: "1170175a71534cac872361156a13e56e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } health_report { overall_health: HEALTHY } } }
W20260619 02:59:39.134737 5192 tablet.cc:2369] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:59:39.145023 5058 tablet.cc:2369] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:39.169642 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.164973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.22.230.3:44543)
0619 02:59:39.165152 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:39.165157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:39.165173 (+ 16us) server_negotiation.cc:407] Connection header received
0619 02:59:39.165341 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:39.165344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:39.165432 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:39.165546 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:59:39.165855 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.166540 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.167926 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.168294 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.168390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.168769 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:59:39.169220 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:59:39.169319 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:39.169412 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 02:59:39.169470 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:59:39.176810 5146 raft_consensus.cc:1261] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Refusing update from remote peer 1170175a71534cac872361156a13e56e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:39.176849 5012 raft_consensus.cc:1261] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Refusing update from remote peer 1170175a71534cac872361156a13e56e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:39.177743 5352 consensus_queue.cc:1035] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:39.178061 5348 consensus_queue.cc:1035] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:39.189258 5357 mvcc.cc:205] Tried to move back new op lower bound from 7298408362699743232 to 7298408362446561280. Current Snapshot: MvccSnapshot[applied={T|T < 7298408362699743232}]
I20260619 02:59:39.211128 4868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:59:39.211390 4868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:59:39.214515 4868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Sending DeleteTablet for 3 replicas of tablet 23ec852ca89e4eb0b2425ec455dbc8d7
I20260619 02:59:39.215535 4992 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ec852ca89e4eb0b2425ec455dbc8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:39 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:51178
I20260619 02:59:39.215612 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ec852ca89e4eb0b2425ec455dbc8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:39 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:51946
I20260619 02:59:39.215620 5259 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ec852ca89e4eb0b2425ec455dbc8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:39 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56684
I20260619 02:59:39.218344 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4929
I20260619 02:59:39.228706 5366 tablet_replica.cc:330] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: stopping tablet replica
I20260619 02:59:39.229967 5367 tablet_replica.cc:330] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: stopping tablet replica
I20260619 02:59:39.230418 5367 raft_consensus.cc:2229] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:59:39.229192 5366 raft_consensus.cc:2229] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:59:39.231786 5366 raft_consensus.cc:2258] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:39.234858 5366 ts_tablet_manager.cc:1905] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:39.231235 5367 raft_consensus.cc:2258] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:39.235558 5365 tablet_replica.cc:330] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: stopping tablet replica
I20260619 02:59:39.235905 5365 raft_consensus.cc:2229] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:59:39.236333 5365 raft_consensus.cc:2258] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:39.237406 5367 ts_tablet_manager.cc:1905] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:39.241127 5365 ts_tablet_manager.cc:1905] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:39.242254 5366 ts_tablet_manager.cc:1918] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:39.242511 5366 log.cc:1196] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/23ec852ca89e4eb0b2425ec455dbc8d7
I20260619 02:59:39.243108 5366 ts_tablet_manager.cc:1939] T 23ec852ca89e4eb0b2425ec455dbc8d7 P f3c29b2b61374bc9be2752eae75d5d64: Deleting consensus metadata
I20260619 02:59:39.244410 5367 ts_tablet_manager.cc:1918] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:39.244577 5367 log.cc:1196] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/23ec852ca89e4eb0b2425ec455dbc8d7
I20260619 02:59:39.244930 4853 catalog_manager.cc:4841] TS f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697): tablet 23ec852ca89e4eb0b2425ec455dbc8d7 (table test-table [id=03c380bb1eda49c5b08182c345c6f83a]) successfully deleted
I20260619 02:59:39.245118 5367 ts_tablet_manager.cc:1939] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 1170175a71534cac872361156a13e56e: Deleting consensus metadata
I20260619 02:59:39.246666 4854 catalog_manager.cc:4841] TS 1170175a71534cac872361156a13e56e (127.22.230.3:44543): tablet 23ec852ca89e4eb0b2425ec455dbc8d7 (table test-table [id=03c380bb1eda49c5b08182c345c6f83a]) successfully deleted
I20260619 02:59:39.247815 5365 ts_tablet_manager.cc:1918] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:39.247964 5365 log.cc:1196] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/23ec852ca89e4eb0b2425ec455dbc8d7
I20260619 02:59:39.248485 5365 ts_tablet_manager.cc:1939] T 23ec852ca89e4eb0b2425ec455dbc8d7 P 14b60302e0fa4031a81b6787ece778f4: Deleting consensus metadata
I20260619 02:59:39.250069 4852 catalog_manager.cc:4841] TS 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557): tablet 23ec852ca89e4eb0b2425ec455dbc8d7 (table test-table [id=03c380bb1eda49c5b08182c345c6f83a]) successfully deleted
Jun 19 02:59:39 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 02:59:39.258899 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.219779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.22.230.1:40557)
0619 02:59:39.219991 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:39.220000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:39.231896 (+ 11896us) server_negotiation.cc:407] Connection header received
0619 02:59:39.232174 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:39.232181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:39.232262 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:39.232342 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:39.236720 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.237791 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.246533 (+ 8742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.247253 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.247363 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.252400 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:39.252414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:39.252431 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:39.252473 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:39.256003 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:39.256679 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:39.256688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:39.256692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:39.256842 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:39.257251 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:39.257256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:39.257259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:39.257737 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:39.258023 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:39.258268 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 02:59:39.258486 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 02:59:39.259490 5052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:39.360158 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4929
I20260619 02:59:39.384809 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5063
Jun 19 02:59:39 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 02:59:39.403501 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.385534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.22.230.2:44697)
0619 02:59:39.385702 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:39.385709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:39.385733 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:59:39.385940 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:39.385946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:39.386018 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:39.386105 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:39.387808 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.388953 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.390832 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.391467 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.391595 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.396693 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:39.396703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:39.396717 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:39.396757 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:39.400515 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:39.401378 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:39.401386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:39.401390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:39.401549 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:39.401937 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:39.401942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:39.401945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:39.402374 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:39.402686 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:39.402948 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 02:59:39.403137 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 02:59:39.404115 5186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:39.489248 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5063
I20260619 02:59:39.510986 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5196
Jun 19 02:59:39 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 02:59:39.528170 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:39.511762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.22.230.3:44543)
0619 02:59:39.511952 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:39.511958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:39.511982 (+ 24us) server_negotiation.cc:407] Connection header received
0619 02:59:39.512047 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:39.512052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:39.512145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:39.512295 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:39.513570 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.514809 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.515964 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:39.516565 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:39.516690 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:39.521100 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:39.521115 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:39.521133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:39.521183 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:39.525179 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:39.525888 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:39.525897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:39.525901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:39.526110 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:39.526481 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:39.526489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:39.526491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:39.527011 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:39.527346 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:39.527532 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 02:59:39.527746 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 02:59:39.528833 5319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:39.645403 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5196
I20260619 02:59:39.665179 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 4836
I20260619 02:59:39.666322 4898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:39.770187 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 4836
I20260619 02:59:39.792248 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45261
--webserver_interface=127.22.230.62
--webserver_port=46681
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:40.104770 5373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:40.105022 5373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:40.105201 5373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:40.112040 5373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:40.112150 5373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:40.112231 5373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:40.112331 5373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:40.112408 5373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:40.121333 5373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:40.122401 5373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:40.123569 5373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:40.146847 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:40.146970 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:40.146921 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:40.152104 5373 server_base.cc:1034] running on GCE node
I20260619 02:59:40.152871 5373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:40.154206 5373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:40.155400 5373 hybrid_clock.cc:648] HybridClock initialized: now 1781837980155363 us; error 25 us; skew 500 ppm
I20260619 02:59:40.155783 5373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:40 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:40.159111 5373 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:40.160724 5373 webserver.cc:460] Webserver started at http://127.22.230.62:46681/ using document root <none> and password file <none>
I20260619 02:59:40.161504 5373 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:40.161659 5373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:40.167276 5373 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 02:59:40.169941 5389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:40.171269 5373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 02:59:40.171465 5373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "9ed0f13cf5fd4324a4f716caebdd10a6"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "2eb00ee538491dbfcab446092bc081b8"
server_key_iv: "ad60ebc7b6598bb743a0553e870e7793"
server_key_version: "encryptionkey@0"
I20260619 02:59:40.172461 5373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:40.203125 5373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:40.206820 5373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:40.207129 5373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:40.225015 5373 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:45261
I20260619 02:59:40.225033 5440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:45261 every 8 connection(s)
I20260619 02:59:40.226406 5373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:40.232628 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5373
I20260619 02:59:40.233179 5441 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:59:40.233549 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:40557
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:45261
--webserver_port=45069
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:59:40.238006 5441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap starting.
I20260619 02:59:40.257678 5441 log.cc:824] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:40.268962 5441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:59:40.269435 5441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap complete.
I20260619 02:59:40.279201 5441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:40.279919 5441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed0f13cf5fd4324a4f716caebdd10a6, State: Initialized, Role: FOLLOWER
I20260619 02:59:40.280506 5441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:40.280722 5441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:59:40.280854 5441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:59:40.281051 5441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 1 FOLLOWER]: Advancing to term 2
I20260619 02:59:40.284634 5441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:40.285151 5441 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed0f13cf5fd4324a4f716caebdd10a6; no voters:
I20260619 02:59:40.285743 5441 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 02:59:40.285853 5445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 02:59:40.287113 5445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 LEADER]: Becoming Leader. State: Replica: 9ed0f13cf5fd4324a4f716caebdd10a6, State: Running, Role: LEADER
I20260619 02:59:40.287683 5445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:40.287909 5441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:40.290318 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed0f13cf5fd4324a4f716caebdd10a6. Latest consensus state: current_term: 2 leader_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } } }
I20260619 02:59:40.290360 5446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } } }
I20260619 02:59:40.290740 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:40.290788 5446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:40.291929 5452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:40.310488 5452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:40.312263 5452 catalog_manager.cc:1259] Loaded cluster ID: b512be3628f94816aa8736aed9f376c9
I20260619 02:59:40.312419 5452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 02:59:40.316236 5452 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 02:59:40.339375 5452 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 02:59:40.354918 5452 catalog_manager.cc:1504] Loading token signing keys...
I20260619 02:59:40.356570 5452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Loaded TSK: 0
I20260619 02:59:40.358047 5452 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 02:59:40.568318 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:40.568602 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:40.568753 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:40.575448 5443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:40.575558 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:40.575733 5443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:59:40.584883 5443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=45069
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:40.586026 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:40.587311 5443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:40.601342 5468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:40.601540 5469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:40.602556 5443 server_base.cc:1034] running on GCE node
W20260619 02:59:40.602835 5471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:40.603390 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:40.604183 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:40.605407 5443 hybrid_clock.cc:648] HybridClock initialized: now 1781837980605352 us; error 50 us; skew 500 ppm
I20260619 02:59:40.605731 5443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:40 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:40.608808 5443 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:59:40.610230 5443 webserver.cc:460] Webserver started at http://127.22.230.1:45069/ using document root <none> and password file <none>
I20260619 02:59:40.610939 5443 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:40.611099 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:40.615816 5443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:59:40.618402 5478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:40.619720 5443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:40.619867 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "14b60302e0fa4031a81b6787ece778f4"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "6e9bfbe568f00f2f99a301e665889dd0"
server_key_iv: "bd72a15441d6922dae488f5ad9955ca2"
server_key_version: "encryptionkey@0"
I20260619 02:59:40.620712 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:40.636200 5443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:40.640338 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:40.640615 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:40.641687 5443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:40.643110 5443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:40.643205 5443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:40.643355 5443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:40.643456 5443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:40.677915 5443 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40557
I20260619 02:59:40.678012 5590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40557 every 8 connection(s)
I20260619 02:59:40.679322 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:59:40.687089 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5443
I20260619 02:59:40.687736 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:44697
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:45261
--webserver_port=42873
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:59:40 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:40.701931 5594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:40.682508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:37785)
0619 02:59:40.683373 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:40.683430 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:40.685123 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:40.686364 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:40.686372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:40.686956 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:40.687564 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:40.687582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:40.689254 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:40.689259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:40.690498 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:40.690511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:40.691560 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:40.691566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:40.691819 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:40.692470 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:40.692492 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:40.694776 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:40.697972 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:40.697980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:40.697993 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:40.698374 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:40.698852 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:40.698858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:40.698864 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:40.699053 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:40.699703 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:40.699710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:40.700010 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 02:59:40.700238 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 02:59:40.700732 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":358,"threads_started":1}
I20260619 02:59:40.701943 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:40.682175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:37785 (local address 127.22.230.62:45261)
0619 02:59:40.682770 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:40.682782 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:40.683541 (+ 759us) server_negotiation.cc:407] Connection header received
0619 02:59:40.685589 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:40.685601 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:40.685902 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:40.686185 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:40.687810 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:40.689099 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:40.690713 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:40.691337 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:40.691528 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:40.695065 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:40.695072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:40.695084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:40.695118 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:40.697824 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:40.698518 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:40.698525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:40.698529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:40.698737 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:40.699168 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:40.699173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:40.699176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:40.699571 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:40.699778 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:40.700442 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 02:59:40.700804 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":248,"threads_started":1}
I20260619 02:59:40.704634 5591 heartbeater.cc:347] Connected to a master server at 127.22.230.62:45261
I20260619 02:59:40.704937 5591 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:40.705591 5591 heartbeater.cc:511] Master 127.22.230.62:45261 requested a full tablet report, sending...
I20260619 02:59:40.707602 5406 ts_manager.cc:194] Registered new tserver with Master: 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557)
I20260619 02:59:40.709043 5406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:37785
W20260619 02:59:40.991559 5595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:40.991837 5595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:40.991997 5595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:40.998654 5595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:40.998786 5595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:40.998965 5595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:59:41.007895 5595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:44697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=42873
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:41.009047 5595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:41.010249 5595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:41.022531 5600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:41.022511 5603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:41.022579 5601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:41.023495 5595 server_base.cc:1034] running on GCE node
I20260619 02:59:41.024116 5595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:41.024964 5595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:41.026178 5595 hybrid_clock.cc:648] HybridClock initialized: now 1781837981026119 us; error 25 us; skew 500 ppm
I20260619 02:59:41.026482 5595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:41 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:41.029456 5595 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:59:41.030786 5595 webserver.cc:460] Webserver started at http://127.22.230.2:42873/ using document root <none> and password file <none>
I20260619 02:59:41.031394 5595 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:41.031538 5595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:41.036134 5595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:59:41.038674 5610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:41.039906 5595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:41.040076 5595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "f3c29b2b61374bc9be2752eae75d5d64"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "6a0e8acb567fb9cd8c367ed4f3471ce0"
server_key_iv: "12328619c1f02282eb709ad60d4a4765"
server_key_version: "encryptionkey@0"
I20260619 02:59:41.041018 5595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:41.067451 5595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:41.070915 5595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:41.071167 5595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:41.071992 5595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:41.073133 5595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:41.073244 5595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:41.073377 5595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:41.073467 5595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:41.104727 5595 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:44697
I20260619 02:59:41.104800 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:44697 every 8 connection(s)
I20260619 02:59:41.105993 5595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:59:41.109966 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5595
I20260619 02:59:41.110671 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:44543
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:45261
--webserver_port=43441
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 02:59:41 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:41.128077 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.108321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:60267 (local address 127.22.230.62:45261)
0619 02:59:41.108488 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.108493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.109533 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 02:59:41.112130 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.112134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.112201 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.112284 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:41.114219 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.115010 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.117458 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.118009 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.118106 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.121501 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:41.121526 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:41.121533 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:41.121574 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:41.125006 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.125619 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.125627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.125633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.125795 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.126184 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.126191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.126197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.126593 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:41.126788 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.127368 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.127635 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:59:41.128631 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.108732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:60267)
0619 02:59:41.109379 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.109413 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.111853 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.112539 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.112553 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.113241 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:41.114052 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.114077 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.115148 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.115157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.117212 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.117228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.118225 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.118234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.118580 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.119237 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:41.119267 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:41.121314 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:41.125115 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:41.125121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:41.125130 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:41.125505 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:41.125911 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:41.125914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:41.125916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:41.126073 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:41.126715 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:41.126720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:41.127012 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.127218 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.127595 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":268,"threads_started":1}
I20260619 02:59:41.130009 5723 heartbeater.cc:347] Connected to a master server at 127.22.230.62:45261
I20260619 02:59:41.130280 5723 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:41.131039 5723 heartbeater.cc:511] Master 127.22.230.62:45261 requested a full tablet report, sending...
I20260619 02:59:41.132385 5406 ts_manager.cc:194] Registered new tserver with Master: f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697)
I20260619 02:59:41.133172 5406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:60267
W20260619 02:59:41.417688 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:41.417944 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:41.418087 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:41.424783 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:41.424943 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:41.425107 5726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:59:41.433933 5726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:44543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=43441
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:41.434980 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:41.436133 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:41.448033 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:41.448040 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:41.448062 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:41.448820 5726 server_base.cc:1034] running on GCE node
I20260619 02:59:41.449364 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:41.450151 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:41.451364 5726 hybrid_clock.cc:648] HybridClock initialized: now 1781837981451326 us; error 47 us; skew 500 ppm
I20260619 02:59:41.451661 5726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:41 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:41.454416 5726 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:59:41.455806 5726 webserver.cc:460] Webserver started at http://127.22.230.3:43441/ using document root <none> and password file <none>
I20260619 02:59:41.456414 5726 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:41.456566 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:41.461299 5726 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 02:59:41.463814 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:41.465073 5726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:41.465238 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "1170175a71534cac872361156a13e56e"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "2546c1c4bed65d5e1b6b7e42ded07d4c"
server_key_iv: "cbe1831dc86e348286468a4b2771b5b8"
server_key_version: "encryptionkey@0"
I20260619 02:59:41.466068 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:41.485219 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:41.488643 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:41.488891 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:41.489717 5726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:41.490890 5726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:41.491016 5726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:41.491137 5726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:41.491231 5726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:41.521539 5726 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:44543
I20260619 02:59:41.521615 5853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:44543 every 8 connection(s)
I20260619 02:59:41.522861 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:59:41.526659 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5726
Jun 19 02:59:41 dist-test-slave-2gms krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837981, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 02:59:41.544523 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.525415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:35989 (local address 127.22.230.62:45261)
0619 02:59:41.525585 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.525590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.526882 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 02:59:41.529057 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.529062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.529165 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.529270 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:41.530653 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.531753 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.533202 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.533839 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.533956 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.537837 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:41.537841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:41.537845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:41.537889 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:41.541207 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.541874 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.541881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.541890 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.542052 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.542709 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.542715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.542718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.542997 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:41.543104 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.543930 (+ 826us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.544179 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 02:59:41.545360 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.527664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.22.230.62:45261)
0619 02:59:41.528130 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.528136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.531160 (+ 3024us) server_negotiation.cc:407] Connection header received
0619 02:59:41.531328 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.531332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.531385 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.531461 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:41.532986 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.534045 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.535938 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.536472 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.536600 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.539423 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:41.539431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:41.539449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:41.539495 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:41.542525 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.543288 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.543292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.543294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.543399 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.543792 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.543799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.543805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.544167 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:41.544423 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.544824 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.545043 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
I20260619 02:59:41.545711 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.525862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:35989)
0619 02:59:41.526686 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.526756 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.528775 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.529493 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.529500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.529954 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 02:59:41.530460 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.530477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.531916 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.531920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.533073 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.533085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.534024 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.534029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.534275 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.534940 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 02:59:41.534973 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 02:59:41.537611 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 02:59:41.541335 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:41.541341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:41.541352 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:41.541746 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:41.542179 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 02:59:41.542182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 02:59:41.542184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 02:59:41.542529 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 02:59:41.543127 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 02:59:41.543134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 02:59:41.543501 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.543811 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.544380 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
I20260619 02:59:41.547032 5854 heartbeater.cc:347] Connected to a master server at 127.22.230.62:45261
I20260619 02:59:41.547325 5854 heartbeater.cc:464] Registering TS with master...
I20260619 02:59:41.548136 5854 heartbeater.cc:511] Master 127.22.230.62:45261 requested a full tablet report, sending...
I20260619 02:59:41.549345 5406 ts_manager.cc:194] Registered new tserver with Master: 1170175a71534cac872361156a13e56e (127.22.230.3:44543)
I20260619 02:59:41.550083 5406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:35989
I20260619 02:59:41.559703 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 02:59:41.565379 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.560505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.22.230.62:45261)
0619 02:59:41.560655 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.560659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.561144 (+ 485us) server_negotiation.cc:407] Connection header received
0619 02:59:41.561293 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.561296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.561409 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.561516 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:59:41.561849 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.562463 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.563744 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.564125 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.564211 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.564589 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:59:41.564981 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:59:41.565075 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.565156 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.565208 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 02:59:41.566087 5406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 02:59:41.573036 5406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 02:59:41.594172 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.584075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.22.230.1:40557)
0619 02:59:41.585149 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.585161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.586282 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 02:59:41.586390 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.586399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.586657 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.586903 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.587921 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.589526 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.591771 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.593083 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.593231 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.593421 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.593708 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.593866 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":762,"threads_started":1}
I20260619 02:59:41.594197 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.583481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51202)
0619 02:59:41.585831 (+ 2350us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.585873 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.586077 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.587225 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.587232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.587264 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.587739 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.587751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.589625 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.589630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.591620 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.591632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.593281 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.593287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.593467 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.593515 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.593656 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.593816 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"mutex_wait_us":22,"thread_start_us":3705,"threads_started":1}
I20260619 02:59:41.595729 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.583518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:44697 (local address 127.0.0.1:51954)
0619 02:59:41.587150 (+ 3632us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.587190 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.587395 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.588030 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.588035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.588059 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.588409 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.588425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.590505 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.590515 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.593087 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.593101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.594904 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.594912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.595096 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.595127 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.595220 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.595335 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3491,"mutex_wait_us":100,"thread_start_us":3343,"threads_started":1}
I20260619 02:59:41.595784 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.583529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.22.230.2:44697)
0619 02:59:41.584670 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.584681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.587288 (+ 2607us) server_negotiation.cc:407] Connection header received
0619 02:59:41.587585 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.587595 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.587837 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.588047 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.588825 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.590349 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.593261 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.594684 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.594818 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.595040 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.595281 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.595448 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":788,"threads_started":1}
I20260619 02:59:41.596154 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.584029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:44543 (local address 127.0.0.1:56710)
0619 02:59:41.587461 (+ 3432us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.587478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.587621 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.588508 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.588513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.588536 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.588891 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.588902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.590358 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.590366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.593088 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.593101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.595332 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.595338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.595499 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.595535 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.595628 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.595744 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3296,"mutex_wait_us":162,"thread_start_us":1348,"threads_started":1}
I20260619 02:59:41.596776 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.584361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.22.230.3:44543)
0619 02:59:41.586628 (+ 2267us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.586639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.587556 (+ 917us) server_negotiation.cc:407] Connection header received
0619 02:59:41.587822 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.587828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.588056 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.588231 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.589113 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.590236 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.593261 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.594836 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.594914 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.595126 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.596360 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.596507 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2076,"thread_start_us":1961,"threads_started":1}
I20260619 02:59:41.598182 5526 tablet_service.cc:1467] Processing CreateTablet for tablet 9823f942617a4811b17a6ef0f3635c63 (DEFAULT_TABLE table=test-table [id=b0ddd2721ad54cf68e8273bde22ce760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:41.599380 5526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9823f942617a4811b17a6ef0f3635c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:41.599320 5658 tablet_service.cc:1467] Processing CreateTablet for tablet 9823f942617a4811b17a6ef0f3635c63 (DEFAULT_TABLE table=test-table [id=b0ddd2721ad54cf68e8273bde22ce760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:41.600615 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9823f942617a4811b17a6ef0f3635c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:41.600230 5789 tablet_service.cc:1467] Processing CreateTablet for tablet 9823f942617a4811b17a6ef0f3635c63 (DEFAULT_TABLE table=test-table [id=b0ddd2721ad54cf68e8273bde22ce760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 02:59:41.601420 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9823f942617a4811b17a6ef0f3635c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 02:59:41.611033 5870 tablet_bootstrap.cc:492] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Bootstrap starting.
I20260619 02:59:41.614008 5870 tablet_bootstrap.cc:654] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:41.615123 5870 log.cc:824] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:41.617622 5870 tablet_bootstrap.cc:492] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: No bootstrap required, opened a new log
I20260619 02:59:41.617893 5870 ts_tablet_manager.cc:1397] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260619 02:59:41.619256 5871 tablet_bootstrap.cc:492] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Bootstrap starting.
I20260619 02:59:41.619575 5872 tablet_bootstrap.cc:492] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Bootstrap starting.
I20260619 02:59:41.621659 5871 tablet_bootstrap.cc:654] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:41.622630 5872 tablet_bootstrap.cc:654] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Neither blocks nor log segments found. Creating new log.
I20260619 02:59:41.622974 5871 log.cc:824] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:41.623828 5872 log.cc:824] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:41.626745 5871 tablet_bootstrap.cc:492] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: No bootstrap required, opened a new log
I20260619 02:59:41.627002 5871 ts_tablet_manager.cc:1397] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 02:59:41.628539 5870 raft_consensus.cc:348] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.629079 5870 raft_consensus.cc:374] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:41.629201 5870 raft_consensus.cc:729] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b60302e0fa4031a81b6787ece778f4, State: Initialized, Role: FOLLOWER
I20260619 02:59:41.629920 5870 consensus_queue.cc:260] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.631107 5591 heartbeater.cc:503] Master 127.22.230.62:45261 was elected leader, sending a full tablet report...
I20260619 02:59:41.631755 5870 ts_tablet_manager.cc:1428] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260619 02:59:41.632825 5872 tablet_bootstrap.cc:492] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: No bootstrap required, opened a new log
I20260619 02:59:41.633133 5872 ts_tablet_manager.cc:1397] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260619 02:59:41.637326 5871 raft_consensus.cc:348] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.637709 5871 raft_consensus.cc:374] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:41.637801 5871 raft_consensus.cc:729] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c29b2b61374bc9be2752eae75d5d64, State: Initialized, Role: FOLLOWER
I20260619 02:59:41.638381 5871 consensus_queue.cc:260] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.639350 5723 heartbeater.cc:503] Master 127.22.230.62:45261 was elected leader, sending a full tablet report...
I20260619 02:59:41.639998 5871 ts_tablet_manager.cc:1428] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260619 02:59:41.642371 5872 raft_consensus.cc:348] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.642813 5872 raft_consensus.cc:374] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 02:59:41.642923 5872 raft_consensus.cc:729] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1170175a71534cac872361156a13e56e, State: Initialized, Role: FOLLOWER
I20260619 02:59:41.643441 5872 consensus_queue.cc:260] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.644480 5854 heartbeater.cc:503] Master 127.22.230.62:45261 was elected leader, sending a full tablet report...
I20260619 02:59:41.644829 5872 ts_tablet_manager.cc:1428] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260619 02:59:41.681425 5592 tablet.cc:2369] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 02:59:41.774431 5855 tablet.cc:2369] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:41.810384 5878 raft_consensus.cc:482] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:59:41.810806 5878 raft_consensus.cc:504] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.812445 5878 leader_election.cc:288] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697), 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557)
I20260619 02:59:41.814174 5877 raft_consensus.cc:482] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 02:59:41.814425 5877 raft_consensus.cc:504] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.820334 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.812640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:44697 (local address 127.22.230.3:60931)
0619 02:59:41.812926 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.812945 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.813067 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.813532 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.813536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.813557 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.813791 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.813802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.815151 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.815156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.816837 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.816855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.819362 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.819370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.819649 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.819699 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.819795 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.819963 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"spinlock_wait_cycles":39808}
I20260619 02:59:41.820655 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.813051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:46763 (local address 127.22.230.1:40557)
0619 02:59:41.813291 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.813296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.813631 (+ 335us) server_negotiation.cc:407] Connection header received
0619 02:59:41.813794 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.813799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.813876 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.813957 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.814423 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.815222 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.817697 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.818976 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.819148 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.819253 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.819954 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.820135 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 02:59:41.821198 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.812727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:60931 (local address 127.22.230.2:44697)
0619 02:59:41.812960 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.812967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.812992 (+ 25us) server_negotiation.cc:407] Connection header received
0619 02:59:41.813296 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.813302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.813403 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.813503 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.813955 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.815014 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.817052 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.819004 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.819170 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.819272 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.820725 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.820897 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 02:59:41.821810 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.812806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.22.230.3:46763)
0619 02:59:41.813534 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.813550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.813678 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.813972 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.813975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.813995 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.814254 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.814260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.815373 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.815378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.817486 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.817503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.819480 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.819488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.819734 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.819782 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.821141 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.821318 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"spinlock_wait_cycles":768,"thread_start_us":440,"threads_started":1}
I20260619 02:59:41.822400 5877 leader_election.cc:288] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557), 1170175a71534cac872361156a13e56e (127.22.230.3:44543)
I20260619 02:59:41.822614 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9823f942617a4811b17a6ef0f3635c63" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c29b2b61374bc9be2752eae75d5d64" is_pre_election: true
I20260619 02:59:41.823076 5678 raft_consensus.cc:2454] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 0.
I20260619 02:59:41.823817 5743 leader_election.cc:302] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1170175a71534cac872361156a13e56e, f3c29b2b61374bc9be2752eae75d5d64; no voters:
I20260619 02:59:41.823694 5546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9823f942617a4811b17a6ef0f3635c63" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b60302e0fa4031a81b6787ece778f4" is_pre_election: true
I20260619 02:59:41.824226 5546 raft_consensus.cc:2454] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 0.
I20260619 02:59:41.824345 5878 raft_consensus.cc:2781] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 02:59:41.824476 5878 raft_consensus.cc:482] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 02:59:41.824560 5878 raft_consensus.cc:3037] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:41.827051 5878 raft_consensus.cc:504] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.827754 5878 leader_election.cc:288] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 election: Requested vote from peers f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697), 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557)
I20260619 02:59:41.828264 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9823f942617a4811b17a6ef0f3635c63" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c29b2b61374bc9be2752eae75d5d64"
I20260619 02:59:41.828248 5546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9823f942617a4811b17a6ef0f3635c63" candidate_uuid: "1170175a71534cac872361156a13e56e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b60302e0fa4031a81b6787ece778f4"
I20260619 02:59:41.828447 5678 raft_consensus.cc:3037] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:41.828459 5546 raft_consensus.cc:3037] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 02:59:41.830881 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.817005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:48225 (local address 127.22.230.3:44543)
0619 02:59:41.817171 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.817176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.823665 (+ 6489us) server_negotiation.cc:407] Connection header received
0619 02:59:41.823883 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.823887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.823954 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.824007 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.824840 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.825962 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.828550 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.829735 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.829842 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.829910 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.830507 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.830672 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 02:59:41.831105 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.816599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:39807 (local address 127.22.230.1:40557)
0619 02:59:41.820902 (+ 4303us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.820910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.822028 (+ 1118us) server_negotiation.cc:407] Connection header received
0619 02:59:41.823686 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.823691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.823777 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.823842 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 02:59:41.824598 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.825626 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.828381 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.829670 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.829821 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.829919 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.830707 (+ 788us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.830830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4192,"thread_start_us":6209,"threads_started":1}
I20260619 02:59:41.831609 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.816433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:44543 (local address 127.22.230.2:48225)
0619 02:59:41.823592 (+ 7159us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.823608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.823744 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.824166 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.824173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.824193 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.824695 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.824707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.826125 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.826131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.828393 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.828405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.830048 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.830058 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.830262 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.830333 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.831029 (+ 696us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.831174 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7041,"mutex_wait_us":156,"thread_start_us":6829,"threads_started":1}
I20260619 02:59:41.831709 5678 raft_consensus.cc:2454] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 1.
I20260619 02:59:41.832015 5546 raft_consensus.cc:2454] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1170175a71534cac872361156a13e56e in term 1.
I20260619 02:59:41.832177 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.816202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.22.230.2:39807)
0619 02:59:41.821938 (+ 5736us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:41.821958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 02:59:41.823525 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 02:59:41.824025 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 02:59:41.824030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 02:59:41.824051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 02:59:41.824434 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.824446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.825786 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.825791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.828183 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 02:59:41.828198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.830109 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.830115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 02:59:41.830514 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.830545 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 02:59:41.831411 (+ 866us) client_negotiation.cc:240] Negotiation successful
0619 02:59:41.831541 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5605,"mutex_wait_us":208,"thread_start_us":5425,"threads_started":1}
I20260619 02:59:41.832219 5743 leader_election.cc:302] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1170175a71534cac872361156a13e56e, f3c29b2b61374bc9be2752eae75d5d64; no voters:
I20260619 02:59:41.832520 5878 raft_consensus.cc:2781] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 02:59:41.833019 5878 raft_consensus.cc:686] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 1 LEADER]: Becoming Leader. State: Replica: 1170175a71534cac872361156a13e56e, State: Running, Role: LEADER
I20260619 02:59:41.833007 5546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9823f942617a4811b17a6ef0f3635c63" candidate_uuid: "f3c29b2b61374bc9be2752eae75d5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b60302e0fa4031a81b6787ece778f4" is_pre_election: true
I20260619 02:59:41.832792 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9823f942617a4811b17a6ef0f3635c63" candidate_uuid: "f3c29b2b61374bc9be2752eae75d5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1170175a71534cac872361156a13e56e" is_pre_election: true
I20260619 02:59:41.833280 5546 raft_consensus.cc:2379] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f3c29b2b61374bc9be2752eae75d5d64 in current term 1: Already voted for candidate 1170175a71534cac872361156a13e56e in this term.
I20260619 02:59:41.834017 5878 consensus_queue.cc:237] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } }
I20260619 02:59:41.837937 5613 leader_election.cc:302] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3c29b2b61374bc9be2752eae75d5d64; no voters: 1170175a71534cac872361156a13e56e, 14b60302e0fa4031a81b6787ece778f4
I20260619 02:59:41.838352 5877 raft_consensus.cc:2726] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 02:59:41.838887 5405 catalog_manager.cc:5495] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1170175a71534cac872361156a13e56e (127.22.230.3). New cstate: current_term: 1 leader_uuid: "1170175a71534cac872361156a13e56e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1170175a71534cac872361156a13e56e" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 44543 } health_report { overall_health: HEALTHY } } }
W20260619 02:59:41.857817 5724 tablet.cc:2369] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 02:59:41.885805 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.881117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.22.230.3:44543)
0619 02:59:41.881267 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.881271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.881289 (+ 18us) server_negotiation.cc:407] Connection header received
0619 02:59:41.881472 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.881475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.881554 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.881664 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 02:59:41.881985 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.882705 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.884008 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.884358 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.884440 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.884838 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 02:59:41.885271 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 02:59:41.885368 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.885450 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.885505 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 02:59:41.893397 5546 raft_consensus.cc:1261] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Refusing update from remote peer 1170175a71534cac872361156a13e56e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:41.893425 5678 raft_consensus.cc:1261] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Refusing update from remote peer 1170175a71534cac872361156a13e56e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 02:59:41.894302 5883 consensus_queue.cc:1035] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b60302e0fa4031a81b6787ece778f4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:41.894670 5878 consensus_queue.cc:1035] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c29b2b61374bc9be2752eae75d5d64" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 02:59:41.905378 5885 mvcc.cc:205] Tried to move back new op lower bound from 7298408373826035712 to 7298408373596184576. Current Snapshot: MvccSnapshot[applied={T|T < 7298408373826035712}]
I20260619 02:59:41.919446 5405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:59:41.919687 5405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 02:59:41.922817 5405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Sending DeleteTablet for 3 replicas of tablet 9823f942617a4811b17a6ef0f3635c63
I20260619 02:59:41.923898 5526 tablet_service.cc:1514] Processing DeleteTablet for tablet 9823f942617a4811b17a6ef0f3635c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:41 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:51202
I20260619 02:59:41.923898 5658 tablet_service.cc:1514] Processing DeleteTablet for tablet 9823f942617a4811b17a6ef0f3635c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:41 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:51954
I20260619 02:59:41.924217 5789 tablet_service.cc:1514] Processing DeleteTablet for tablet 9823f942617a4811b17a6ef0f3635c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 02:59:41 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56710
I20260619 02:59:41.924737 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5443
I20260619 02:59:41.938998 5897 tablet_replica.cc:330] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: stopping tablet replica
I20260619 02:59:41.939410 5897 raft_consensus.cc:2229] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 1 LEADER]: Raft consensus shutting down.
I20260619 02:59:41.940068 5897 raft_consensus.cc:2258] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:41.942310 5897 ts_tablet_manager.cc:1905] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:41.942637 5896 tablet_replica.cc:330] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: stopping tablet replica
I20260619 02:59:41.943967 5895 tablet_replica.cc:330] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: stopping tablet replica
I20260619 02:59:41.943941 5896 raft_consensus.cc:2229] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:59:41.944424 5896 raft_consensus.cc:2258] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:41.944407 5895 raft_consensus.cc:2229] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 02:59:41.944856 5895 raft_consensus.cc:2258] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 02:59:41.946620 5896 ts_tablet_manager.cc:1905] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:41.946815 5895 ts_tablet_manager.cc:1905] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:41.952473 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:41.925500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.22.230.1:40557)
0619 02:59:41.925687 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:41.925696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:41.925781 (+ 85us) server_negotiation.cc:407] Connection header received
0619 02:59:41.925964 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:41.925971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:41.926089 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:41.926203 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:41.927919 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.929414 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.931369 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:41.932195 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:41.932298 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:41.935570 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:41.935584 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:41.935598 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:41.935646 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:41.939965 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.947080 (+ 7115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.947089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.947092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.947250 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:41.947727 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:41.947732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:41.947736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:41.948188 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:41.948505 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:41.951856 (+ 3351us) server_negotiation.cc:299] Negotiation successful
0619 02:59:41.952104 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 02:59:41.953467 5586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:41.953608 5896 ts_tablet_manager.cc:1918] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:41.953795 5896 log.cc:1196] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/9823f942617a4811b17a6ef0f3635c63
I20260619 02:59:41.954375 5896 ts_tablet_manager.cc:1939] T 9823f942617a4811b17a6ef0f3635c63 P f3c29b2b61374bc9be2752eae75d5d64: Deleting consensus metadata
I20260619 02:59:41.956197 5391 catalog_manager.cc:4841] TS f3c29b2b61374bc9be2752eae75d5d64 (127.22.230.2:44697): tablet 9823f942617a4811b17a6ef0f3635c63 (table test-table [id=b0ddd2721ad54cf68e8273bde22ce760]) successfully deleted
I20260619 02:59:41.957499 5897 ts_tablet_manager.cc:1918] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:41.957660 5897 log.cc:1196] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/9823f942617a4811b17a6ef0f3635c63
I20260619 02:59:41.958220 5897 ts_tablet_manager.cc:1939] T 9823f942617a4811b17a6ef0f3635c63 P 1170175a71534cac872361156a13e56e: Deleting consensus metadata
I20260619 02:59:41.959856 5392 catalog_manager.cc:4841] TS 1170175a71534cac872361156a13e56e (127.22.230.3:44543): tablet 9823f942617a4811b17a6ef0f3635c63 (table test-table [id=b0ddd2721ad54cf68e8273bde22ce760]) successfully deleted
I20260619 02:59:42.041325 5895 ts_tablet_manager.cc:1918] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:42.041599 5895 log.cc:1196] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/9823f942617a4811b17a6ef0f3635c63
I20260619 02:59:42.041981 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5443
I20260619 02:59:42.064108 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5595
W20260619 02:59:42.064502 5390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.230.1:40557 (error 108)
I20260619 02:59:42.077876 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.077418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51214)
0619 02:59:42.077583 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.077702 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 02:59:42.078307 5390 catalog_manager.cc:4584] TS 14b60302e0fa4031a81b6787ece778f4 (127.22.230.1:40557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9823f942617a4811b17a6ef0f3635c63: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
I20260619 02:59:42.086872 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.068865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.22.230.2:44697)
0619 02:59:42.070004 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:42.070037 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:42.070059 (+ 22us) server_negotiation.cc:407] Connection header received
0619 02:59:42.070147 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:42.070153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:42.070296 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:42.070404 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:42.072880 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:42.073990 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:42.076802 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:42.077504 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:42.077628 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:42.081190 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:42.081204 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:42.081219 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:42.081255 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:42.084086 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:42.084870 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:42.084878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:42.084880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:42.085005 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:42.085394 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:42.085397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:42.085399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:42.085718 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:42.085930 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:42.086264 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 02:59:42.086451 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851}
I20260619 02:59:42.087493 5718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:42.131778 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.131342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51222)
0619 02:59:42.131548 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.131640 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 02:59:42.132838 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.132494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51224)
0619 02:59:42.132642 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.132726 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:59:42.193614 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.193045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51238)
0619 02:59:42.193298 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.193407 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 02:59:42.194794 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.194349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51254)
0619 02:59:42.194519 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.194643 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 02:59:42.220188 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5595
I20260619 02:59:42.237919 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5726
I20260619 02:59:42.252130 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.238688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.22.230.3:44543)
0619 02:59:42.238914 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:42.238921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:42.238983 (+ 62us) server_negotiation.cc:407] Connection header received
0619 02:59:42.239206 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:42.239210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:42.239285 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:42.239399 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:42.240690 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:42.241441 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:42.242890 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:42.243352 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:42.243496 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:42.246093 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:42.246107 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:42.246124 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:42.246172 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:42.249297 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:42.250015 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:42.250022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:42.250024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:42.250149 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:42.250524 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:42.250529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:42.250533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:42.250974 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:42.251302 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:42.251574 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 02:59:42.251808 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 02:59:42.252784 5849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:42.276176 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.275616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51264)
0619 02:59:42.275802 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.275933 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 02:59:42.277235 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.276871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51278)
0619 02:59:42.277056 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.277133 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:59:42.372233 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5726
I20260619 02:59:42.392325 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5373
I20260619 02:59:42.393427 5436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:42.503175 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:42.502516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40557 (local address 127.0.0.1:51282)
0619 02:59:42.502731 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:42.502817 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:40557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 02:59:42.503754 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5373
I20260619 02:59:42.525698 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45261
--webserver_interface=127.22.230.62
--webserver_port=46681
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:42.844172 5903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:42.844478 5903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:42.844645 5903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:42.851716 5903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 02:59:42.851861 5903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:42.851969 5903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:42.852056 5903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 02:59:42.852123 5903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 02:59:42.862057 5903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:42.863348 5903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:42.864607 5903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:42.876281 5908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:42.876346 5911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:42.876335 5909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:42.876981 5903 server_base.cc:1034] running on GCE node
I20260619 02:59:42.877723 5903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:42.879074 5903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:42.880263 5903 hybrid_clock.cc:648] HybridClock initialized: now 1781837982880213 us; error 62 us; skew 500 ppm
I20260619 02:59:42.880601 5903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:42 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837982, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:42.887033 5903 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 02:59:42.888382 5903 webserver.cc:460] Webserver started at http://127.22.230.62:46681/ using document root <none> and password file <none>
I20260619 02:59:42.889029 5903 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:42.889180 5903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:42.894217 5903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 02:59:42.896910 5918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:42.898201 5903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 02:59:42.898376 5903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "9ed0f13cf5fd4324a4f716caebdd10a6"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "2eb00ee538491dbfcab446092bc081b8"
server_key_iv: "ad60ebc7b6598bb743a0553e870e7793"
server_key_version: "encryptionkey@0"
I20260619 02:59:42.899369 5903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:42.947896 5903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:42.951845 5903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:42.952118 5903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:42.970328 5903 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:45261
I20260619 02:59:42.970345 5969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:45261 every 8 connection(s)
I20260619 02:59:42.971794 5903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 02:59:42.978271 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5903
I20260619 02:59:42.978753 5970 sys_catalog.cc:263] Verifying existing consensus state
I20260619 02:59:42.979291 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:40557
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:45261
--webserver_port=45069
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 02:59:42.983844 5970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap starting.
I20260619 02:59:43.005182 5970 log.cc:824] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Log is configured to *not* fsync() on all Append() calls
I20260619 02:59:43.021162 5970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 02:59:43.021718 5970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6: Bootstrap complete.
I20260619 02:59:43.030808 5970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:43.031657 5970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed0f13cf5fd4324a4f716caebdd10a6, State: Initialized, Role: FOLLOWER
I20260619 02:59:43.032353 5970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:43.032575 5970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 02:59:43.032684 5970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 02:59:43.032877 5970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 2 FOLLOWER]: Advancing to term 3
I20260619 02:59:43.036599 5970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:43.037102 5970 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed0f13cf5fd4324a4f716caebdd10a6; no voters:
I20260619 02:59:43.037815 5970 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 02:59:43.038002 5974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 02:59:43.039148 5974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [term 3 LEADER]: Becoming Leader. State: Replica: 9ed0f13cf5fd4324a4f716caebdd10a6, State: Running, Role: LEADER
I20260619 02:59:43.039669 5974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } }
I20260619 02:59:43.040016 5970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 02:59:43.042161 5976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed0f13cf5fd4324a4f716caebdd10a6. Latest consensus state: current_term: 3 leader_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } } }
I20260619 02:59:43.042692 5976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:43.042258 5975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed0f13cf5fd4324a4f716caebdd10a6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45261 } } }
I20260619 02:59:43.043316 5975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed0f13cf5fd4324a4f716caebdd10a6 [sys.catalog]: This master's current role is: LEADER
I20260619 02:59:43.056020 5979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 02:59:43.064680 5979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 02:59:43.066485 5979 catalog_manager.cc:1259] Loaded cluster ID: b512be3628f94816aa8736aed9f376c9
I20260619 02:59:43.066673 5979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 02:59:43.068955 5979 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe95f6e39ad google::LogMessage::Fail() at ??:0
@ 0x7fe95f6e688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe95f6e34de google::LogMessage::Flush() at ??:0
@ 0x7fe95f6e4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe975116512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe975113976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe97519b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe97519b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe96232f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe96233291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe962332799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe96230deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe9655e76db start_thread at ??:0
@ 0x7fe95e57d61f clone at ??:0
W20260619 02:59:43.369257 5972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:43.369510 5972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:43.369602 5972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:43.376616 5972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:43.376727 5972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:43.376902 5972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 02:59:43.386746 5972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:40557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=45069
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:43.387951 5972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:43.389257 5972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:43.402091 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:43.402122 6012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:43.402632 6014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:43.403153 5972 server_base.cc:1034] running on GCE node
I20260619 02:59:43.403699 5972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:43.404526 5972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:43.405712 5972 hybrid_clock.cc:648] HybridClock initialized: now 1781837983405679 us; error 30 us; skew 500 ppm
I20260619 02:59:43.406008 5972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:43 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:43.409001 5972 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 02:59:43.410429 5972 webserver.cc:460] Webserver started at http://127.22.230.1:45069/ using document root <none> and password file <none>
I20260619 02:59:43.411120 5972 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:43.411275 5972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:43.416155 5972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:59:43.418850 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:43.420112 5972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:43.420271 5972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "14b60302e0fa4031a81b6787ece778f4"
format_stamp: "Formatted at 2026-06-19 02:59:37 on dist-test-slave-2gms"
server_key: "6e9bfbe568f00f2f99a301e665889dd0"
server_key_iv: "bd72a15441d6922dae488f5ad9955ca2"
server_key_version: "encryptionkey@0"
I20260619 02:59:43.421190 5972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:43.434986 5972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:43.438658 5972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:43.438913 5972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:43.439797 5972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:43.441465 6028 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 02:59:43.447006 6028 ts_tablet_manager.cc:1852] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 02:59:43.447966 6028 ts_tablet_manager.cc:1905] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 02:59:43.454803 6028 ts_tablet_manager.cc:1918] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 02:59:43.455005 6028 log.cc:1196] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/9823f942617a4811b17a6ef0f3635c63
I20260619 02:59:43.455499 6028 ts_tablet_manager.cc:1939] T 9823f942617a4811b17a6ef0f3635c63 P 14b60302e0fa4031a81b6787ece778f4: Deleting consensus metadata
I20260619 02:59:43.455874 5972 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 02:59:43.456040 5972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260619 02:59:43.456171 5972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:43.456266 5972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:43.489269 5972 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40557
I20260619 02:59:43.489296 6134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40557 every 8 connection(s)
I20260619 02:59:43.490537 5972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 02:59:43.495136 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:43.493370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:52707)
0619 02:59:43.494037 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:43.494268 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":253,"threads_started":1}
W20260619 02:59:43.495705 6135 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:45261 (0 consecutive failures): Network error: Failed to ping master at 127.22.230.62:45261: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
I20260619 02:59:43.496451 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:43.496148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:43003)
0619 02:59:43.496282 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:43.496351 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 02:59:43.497184 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 5972
I20260619 02:59:43.497321 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:43.496925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:53595)
0619 02:59:43.497076 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:43.497169 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 02:59:43.497510 6135 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:59:43.497942 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:44697
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:45261
--webserver_port=42873
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:43.809368 6138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:43.809631 6138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:43.809813 6138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:43.816869 6138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:43.816992 6138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:43.817175 6138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 02:59:43.826258 6138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:44697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=42873
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:43.827450 6138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:43.828701 6138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:43.841571 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:43.841542 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:43.841634 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:43.842317 6138 server_base.cc:1034] running on GCE node
I20260619 02:59:43.842898 6138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:43.843750 6138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:43.844980 6138 hybrid_clock.cc:648] HybridClock initialized: now 1781837983844944 us; error 51 us; skew 500 ppm
I20260619 02:59:43.845306 6138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:43 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837983, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:43.848337 6138 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 02:59:43.849891 6138 webserver.cc:460] Webserver started at http://127.22.230.2:42873/ using document root <none> and password file <none>
I20260619 02:59:43.850559 6138 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:43.850771 6138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:43.855775 6138 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 02:59:43.858283 6153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:43.859623 6138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:43.859814 6138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "f3c29b2b61374bc9be2752eae75d5d64"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "6a0e8acb567fb9cd8c367ed4f3471ce0"
server_key_iv: "12328619c1f02282eb709ad60d4a4765"
server_key_version: "encryptionkey@0"
I20260619 02:59:43.860847 6138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:43.882332 6138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:43.885970 6138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:43.886178 6138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:43.887075 6138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:43.888298 6138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:43.888381 6138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:43.888465 6138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:43.888520 6138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:43.921725 6138 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:44697
I20260619 02:59:43.921948 6265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:44697 every 8 connection(s)
I20260619 02:59:43.923214 6138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 02:59:43.927974 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:43.926140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:59695)
0619 02:59:43.926935 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:43.927188 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":251,"threads_started":1}
W20260619 02:59:43.928668 6266 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:45261 (0 consecutive failures): Network error: Failed to ping master at 127.22.230.62:45261: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
I20260619 02:59:43.929409 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:43.929074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:36137)
0619 02:59:43.929233 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:43.929310 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 02:59:43.929876 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 6138
I20260619 02:59:43.930305 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:43.930007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:43841)
0619 02:59:43.930126 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:43.930201 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:59:43.930555 6266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:59:43.930613 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:44543
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:45261
--webserver_port=43441
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 02:59:44.248413 6269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 02:59:44.248711 6269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 02:59:44.248864 6269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 02:59:44.255751 6269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 02:59:44.255887 6269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 02:59:44.256069 6269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 02:59:44.265591 6269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:44543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=43441
--tserver_master_addrs=127.22.230.62:45261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 02:59:44.266795 6269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 02:59:44.268014 6269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 02:59:44.280478 6275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:44.280450 6277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 02:59:44.280504 6274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 02:59:44.281996 6269 server_base.cc:1034] running on GCE node
I20260619 02:59:44.282537 6269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 02:59:44.283424 6269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 02:59:44.284652 6269 hybrid_clock.cc:648] HybridClock initialized: now 1781837984284611 us; error 52 us; skew 500 ppm
I20260619 02:59:44.284946 6269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 02:59:44 dist-test-slave-2gms krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781837984, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 02:59:44.287935 6269 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 02:59:44.289247 6269 webserver.cc:460] Webserver started at http://127.22.230.3:43441/ using document root <none> and password file <none>
I20260619 02:59:44.289888 6269 fs_manager.cc:359] Metadata directory not provided
I20260619 02:59:44.290027 6269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 02:59:44.294981 6269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 02:59:44.297590 6284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:44.298873 6269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 02:59:44.299049 6269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "1170175a71534cac872361156a13e56e"
format_stamp: "Formatted at 2026-06-19 02:59:38 on dist-test-slave-2gms"
server_key: "2546c1c4bed65d5e1b6b7e42ded07d4c"
server_key_iv: "cbe1831dc86e348286468a4b2771b5b8"
server_key_version: "encryptionkey@0"
I20260619 02:59:44.300055 6269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 02:59:44.338871 6269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 02:59:44.342489 6269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 02:59:44.342768 6269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 02:59:44.343654 6269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 02:59:44.344878 6269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 02:59:44.345000 6269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:44.345134 6269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 02:59:44.345220 6269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 02:59:44.377254 6269 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:44543
I20260619 02:59:44.377337 6396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:44543 every 8 connection(s)
I20260619 02:59:44.378690 6269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 02:59:44.382047 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 6269
I20260619 02:59:44.383517 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:44.381367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:42161)
0619 02:59:44.382112 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:44.382392 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":315,"threads_started":1}
W20260619 02:59:44.384312 6397 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:45261 (0 consecutive failures): Network error: Failed to ping master at 127.22.230.62:45261: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
I20260619 02:59:44.384996 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:44.384710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:48819)
0619 02:59:44.384844 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:44.384906 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 02:59:44.385741 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:44.385475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:55759)
0619 02:59:44.385583 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:44.385647 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 02:59:44.385972 6397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 02:59:44.498898 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:44.498112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:59497)
0619 02:59:44.498531 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:44.498665 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":199,"threads_started":1}
I20260619 02:59:44.931846 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:44.931105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:41775)
0619 02:59:44.931547 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:44.931646 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":209,"threads_started":1}
I20260619 02:59:45.387269 6403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:45.386403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:36127)
0619 02:59:45.386962 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:45.387067 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":332,"threads_started":1}
I20260619 02:59:45.500350 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:45.499628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:59851)
0619 02:59:45.500104 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:45.500182 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":276,"threads_started":1}
I20260619 02:59:45.933431 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:45.932758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:36211)
0619 02:59:45.933203 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:45.933278 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":242,"threads_started":1}
I20260619 02:59:46.388746 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:46.388040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:45499)
0619 02:59:46.388528 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:46.388602 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
I20260619 02:59:46.501804 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:46.501149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:55957)
0619 02:59:46.501568 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:46.501645 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260619 02:59:46.935091 6408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:46.934346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:44655)
0619 02:59:46.934821 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:46.934910 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260619 02:59:47.390172 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:47.389514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:56403)
0619 02:59:47.389925 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:47.390017 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":208,"threads_started":1}
I20260619 02:59:47.503495 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:47.502781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:58699)
0619 02:59:47.503242 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:47.503319 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":228,"threads_started":1}
I20260619 02:59:47.936689 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:47.935885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:56987)
0619 02:59:47.936442 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:47.936517 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":350,"threads_started":1}
I20260619 02:59:48.391607 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:48.390967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:33903)
0619 02:59:48.391382 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:48.391455 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":211,"threads_started":1}
I20260619 02:59:48.505033 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:48.504366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:33147)
0619 02:59:48.504806 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:48.504875 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":246,"threads_started":1}
I20260619 02:59:48.938186 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:48.937554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:48347)
0619 02:59:48.937966 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:48.938038 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":204,"threads_started":1}
I20260619 02:59:49.393164 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:49.392565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:36295)
0619 02:59:49.392954 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:49.393022 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":216,"threads_started":1}
I20260619 02:59:49.506500 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:49.505790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:58419)
0619 02:59:49.506283 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:49.506354 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":291,"threads_started":1}
I20260619 02:59:49.939631 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:49.939002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:52775)
0619 02:59:49.939401 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:49.939482 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":202,"threads_started":1}
I20260619 02:59:50.394610 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:50.393997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:48355)
0619 02:59:50.394380 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:50.394457 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":199,"threads_started":1}
I20260619 02:59:50.508064 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:50.507326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:42485)
0619 02:59:50.507772 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:50.507854 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":214,"threads_started":1}
I20260619 02:59:50.941133 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:50.940423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:53393)
0619 02:59:50.940850 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:50.940932 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":177,"threads_started":1}
I20260619 02:59:51.396138 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:51.395406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:37359)
0619 02:59:51.395871 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:51.395949 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":205,"threads_started":1}
I20260619 02:59:51.509478 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:51.508848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:58191)
0619 02:59:51.509251 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:51.509327 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":216,"threads_started":1}
I20260619 02:59:51.942948 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:51.942130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:49489)
0619 02:59:51.942690 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:51.942796 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":311,"threads_started":1}
I20260619 02:59:52.397688 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:52.397089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:59913)
0619 02:59:52.397479 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:52.397555 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":204,"threads_started":1}
I20260619 02:59:52.511101 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:52.510380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:50973)
0619 02:59:52.510864 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:52.510954 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260619 02:59:52.944766 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:52.943933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:60411)
0619 02:59:52.944523 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:52.944610 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":326,"threads_started":1}
I20260619 02:59:53.399622 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:53.398821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:46691)
0619 02:59:53.399313 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:53.399461 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":220,"threads_started":1}
I20260619 02:59:53.512627 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:53.511858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:49417)
0619 02:59:53.512408 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:53.512482 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":350,"threads_started":1}
I20260619 02:59:53.946285 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:53.945586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:44037)
0619 02:59:53.946047 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:53.946139 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":281,"threads_started":1}
I20260619 02:59:54.401234 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:54.400537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:57569)
0619 02:59:54.401023 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:54.401095 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":299,"threads_started":1}
I20260619 02:59:54.514102 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:54.513470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:36119)
0619 02:59:54.513866 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:54.513944 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":221,"threads_started":1}
I20260619 02:59:54.947798 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:54.947143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:38223)
0619 02:59:54.947544 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:54.947630 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":214,"threads_started":1}
I20260619 02:59:55.402719 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:55.402002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:55925)
0619 02:59:55.402401 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:55.402483 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":206,"threads_started":1}
I20260619 02:59:55.515571 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:55.514926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:35541)
0619 02:59:55.515339 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:55.515413 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":216,"threads_started":1}
I20260619 02:59:55.949369 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:55.948731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:51601)
0619 02:59:55.949136 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:55.949210 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
I20260619 02:59:56.404044 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:56.403467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:39427)
0619 02:59:56.403843 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:56.403914 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":198,"threads_started":1}
I20260619 02:59:56.517115 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:56.516450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:48841)
0619 02:59:56.516876 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:56.516962 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260619 02:59:56.951050 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:56.950316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:35367)
0619 02:59:56.950783 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:56.950867 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":204,"threads_started":1}
I20260619 02:59:57.405536 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:57.404823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:55439)
0619 02:59:57.405281 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:57.405365 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":229,"threads_started":1}
I20260619 02:59:57.518723 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:57.517980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:60903)
0619 02:59:57.518401 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:57.518493 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":204,"threads_started":1}
I20260619 02:59:57.952564 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:57.951940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:38141)
0619 02:59:57.952329 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:57.952402 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":193,"threads_started":1}
I20260619 02:59:58.407096 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:58.406418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:36695)
0619 02:59:58.406855 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:58.406934 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260619 02:59:58.520329 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:58.519665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.1:58227)
0619 02:59:58.520085 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:58.520169 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":223,"threads_started":1}
I20260619 02:59:58.954476 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:58.953742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.2:43925)
0619 02:59:58.954145 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:58.954292 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":209,"threads_started":1}
I20260619 02:59:59.385504 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 5972
I20260619 02:59:59.401532 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:59.386542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.22.230.1:40557)
0619 02:59:59.387166 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:59.387187 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:59.387232 (+ 45us) server_negotiation.cc:407] Connection header received
0619 02:59:59.388963 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:59.388972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:59.389380 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:59.389657 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:59.390977 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:59.392116 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:59.393236 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:59.393827 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:59.393959 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:59.396258 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:59.396264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:59.396275 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:59.396302 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:59.398791 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:59.399388 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:59.399394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:59.399399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:59.399564 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:59.399883 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:59.399886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:59.399888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:59.400340 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:59.400553 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:59.400876 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 02:59:59.401264 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":270,"threads_started":1}
I20260619 02:59:59.402812 6130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:59.408677 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:59.408063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:45261 (local address 127.22.230.3:54647)
0619 02:59:59.408447 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 02:59:59.408526 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":198,"threads_started":1}
I20260619 02:59:59.483598 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 5972
I20260619 02:59:59.501428 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 6138
I20260619 02:59:59.518669 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:59.502291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.22.230.2:44697)
0619 02:59:59.502864 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:59.502881 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:59.502928 (+ 47us) server_negotiation.cc:407] Connection header received
0619 02:59:59.504594 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:59.504603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:59.504990 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:59.505323 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:59.506665 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:59.507875 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:59.509062 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:59.509707 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:59.509903 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:59.512480 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:59.512487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:59.512499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:59.512560 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:59.515590 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:59.516241 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:59.516247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:59.516253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:59.516412 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:59.516797 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:59.516802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:59.516805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:59.517256 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:59.517557 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:59.517971 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 02:59:59.518339 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":227,"threads_started":1}
I20260619 02:59:59.519938 6261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:59.598222 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 6138
I20260619 02:59:59.616338 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 6269
I20260619 02:59:59.634198 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 02:59:59.617336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.22.230.3:44543)
0619 02:59:59.617880 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 02:59:59.617894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 02:59:59.617933 (+ 39us) server_negotiation.cc:407] Connection header received
0619 02:59:59.619591 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 02:59:59.619599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 02:59:59.620041 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 02:59:59.620399 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 02:59:59.621774 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:59.623109 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:59.624316 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 02:59:59.625043 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 02:59:59.625255 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 02:59:59.627920 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 02:59:59.627928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 02:59:59.627959 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 02:59:59.627994 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 02:59:59.631017 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:59.631692 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:59.631700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:59.631715 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:59.631873 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 02:59:59.632272 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 02:59:59.632278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 02:59:59.632281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 02:59:59.632770 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 02:59:59.633087 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 02:59:59.633527 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 02:59:59.633907 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":255,"threads_started":1}
I20260619 02:59:59.635552 6392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 02:59:59.718050 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 6269
2026-06-19T02:59:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24100 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 02:59:59 dist-test-slave-2gms krb5kdc[6454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 02:59:59 dist-test-slave-2gms krb5kdc[6454](info): set up 2 sockets
Jun 19 02:59:59 dist-test-slave-2gms krb5kdc[6454](info): commencing operation
krb5kdc: starting...
W20260619 03:00:00.807250 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:00:00 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:00:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:00:01.025465 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36975
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:36975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:00:01.337376 6467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:01.337738 6467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:01.337913 6467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:01.344825 6467 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 03:00:01.344949 6467 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 03:00:01.345043 6467 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 03:00:01.345139 6467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:00:01.345216 6467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:01.345316 6467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:01.345382 6467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:00:01.345455 6467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:00:01.354823 6467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:36975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:01.356002 6467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:01.357242 6467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:01.369488 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:01.369405 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:01.369489 6472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:01.369726 6467 server_base.cc:1034] running on GCE node
I20260619 03:00:01.370833 6467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:01.372076 6467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:01.373294 6467 hybrid_clock.cc:648] HybridClock initialized: now 1781838001373227 us; error 38 us; skew 500 ppm
I20260619 03:00:01.373636 6467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:01 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:01.376701 6467 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:00:01.377990 6467 webserver.cc:460] Webserver started at http://127.22.230.62:42781/ using document root <none> and password file <none>
I20260619 03:00:01.378711 6467 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:01.378835 6467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:01.379170 6467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:00:01.381970 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "541d6c93a16b44feb124cc41a20a5571"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "28e77a22cdfad18a9eba9582e6e67faa"
server_key_iv: "5396185b3644eafa11bdc1cf099a5138"
server_key_version: "encryptionkey@0"
I20260619 03:00:01.382781 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "541d6c93a16b44feb124cc41a20a5571"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "28e77a22cdfad18a9eba9582e6e67faa"
server_key_iv: "5396185b3644eafa11bdc1cf099a5138"
server_key_version: "encryptionkey@0"
I20260619 03:00:01.387755 6467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 03:00:01.391162 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:01.392473 6467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 03:00:01.392653 6467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "541d6c93a16b44feb124cc41a20a5571"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "28e77a22cdfad18a9eba9582e6e67faa"
server_key_iv: "5396185b3644eafa11bdc1cf099a5138"
server_key_version: "encryptionkey@0"
I20260619 03:00:01.392861 6467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:01.427778 6467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:01.432228 6467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:01.432507 6467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:01.450809 6467 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:36975
I20260619 03:00:01.450835 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:36975 every 8 connection(s)
I20260619 03:00:01.452136 6467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:00:01.454603 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 6467
I20260619 03:00:01.454829 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:00:01.455451 23448 external_mini_cluster.cc:1428] Setting key 02cd5008e7d0fba0b490bfa8cccc5580
I20260619 03:00:01.455915 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:01.469729 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap starting.
Jun 19 03:00:01 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:01.473232 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:01.474291 6534 log.cc:824] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:01.477514 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: No bootstrap required, opened a new log
I20260619 03:00:01.479852 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:01.457407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.22.230.62:36975)
0619 03:00:01.458208 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:01.458222 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:01.458270 (+ 48us) server_negotiation.cc:407] Connection header received
0619 03:00:01.460571 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:01.460597 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:01.461100 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:01.461520 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:01.462867 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:01.464838 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:01.466296 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:01.467421 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:01.467599 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:01.472169 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:01.472177 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:01.472190 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:01.472225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:01.475451 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:01.476152 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:01.476164 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:01.476170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:01.476379 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:01.476745 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:01.476750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:01.476753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:01.477415 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:01.477687 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:01.478107 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 03:00:01.478525 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":316,"threads_started":1}
I20260619 03:00:01.485948 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:01.486255 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:01.486334 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541d6c93a16b44feb124cc41a20a5571, State: Initialized, Role: FOLLOWER
I20260619 03:00:01.486914 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:01.487170 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:00:01.487318 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:00:01.487501 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:01.489671 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:01.490092 6534 leader_election.cc:302] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541d6c93a16b44feb124cc41a20a5571; no voters:
I20260619 03:00:01.490757 6534 leader_election.cc:288] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:00:01.490913 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:00:01.491688 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 LEADER]: Becoming Leader. State: Replica: 541d6c93a16b44feb124cc41a20a5571, State: Running, Role: LEADER
I20260619 03:00:01.492194 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:01.492795 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:00:01.494830 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "541d6c93a16b44feb124cc41a20a5571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } } }
I20260619 03:00:01.495147 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: This master's current role is: LEADER
I20260619 03:00:01.495611 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541d6c93a16b44feb124cc41a20a5571. Latest consensus state: current_term: 1 leader_uuid: "541d6c93a16b44feb124cc41a20a5571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } } }
I20260619 03:00:01.495914 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: This master's current role is: LEADER
I20260619 03:00:01.496037 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:00:01.500711 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:00:01.509398 6545 catalog_manager.cc:1347] Generated new cluster ID: 464a29180a7a44ceb9c8adf70f7bd065
I20260619 03:00:01.509516 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:00:01.541100 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:00:01.542500 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:00:01.549117 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Generated new TSK 0
I20260619 03:00:01.549891 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 03:00:01.609443 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36975
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:00:01.925773 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:01.926034 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:01.926188 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:01.933264 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:01.933360 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:01.933544 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:00:01.943192 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:01.944404 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:01.945621 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:01.958216 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:01.958187 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:01.958295 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:01.959172 6560 server_base.cc:1034] running on GCE node
I20260619 03:00:01.959779 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:01.960630 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:01.961859 6560 hybrid_clock.cc:648] HybridClock initialized: now 1781838001961817 us; error 51 us; skew 500 ppm
I20260619 03:00:01.962157 6560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:01 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:01.969070 6560 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 03:00:01.970438 6560 webserver.cc:460] Webserver started at http://127.22.230.1:44133/ using document root <none> and password file <none>
I20260619 03:00:01.971119 6560 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:01.971264 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:01.971554 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:00:01.974264 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "37289167011ebd386fda7a0013f9885c"
server_key_iv: "7f1a75c4fd23364be69adc676918fd78"
server_key_version: "encryptionkey@0"
I20260619 03:00:01.975003 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "37289167011ebd386fda7a0013f9885c"
server_key_iv: "7f1a75c4fd23364be69adc676918fd78"
server_key_version: "encryptionkey@0"
I20260619 03:00:01.979770 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:00:01.983062 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:01.984217 6560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 03:00:01.984386 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "37289167011ebd386fda7a0013f9885c"
server_key_iv: "7f1a75c4fd23364be69adc676918fd78"
server_key_version: "encryptionkey@0"
I20260619 03:00:01.984563 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:02.014467 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:02.018121 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:02.018405 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:02.019304 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:02.020539 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:02.020642 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.020789 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:02.020876 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.053143 6560 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40605
I20260619 03:00:02.053225 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40605 every 8 connection(s)
I20260619 03:00:02.054369 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:00:02.063277 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 6560
I20260619 03:00:02.063493 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:00:02.064396 23448 external_mini_cluster.cc:1428] Setting key 1d02bb4d2b34971245f0502a39d3a276
Jun 19 03:00:02 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:02.076638 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:02.056783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:53447 (local address 127.22.230.62:36975)
0619 03:00:02.057214 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:02.057221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:02.058181 (+ 960us) server_negotiation.cc:407] Connection header received
0619 03:00:02.060934 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:02.060942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:02.061041 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:02.061167 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:02.063373 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.064504 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.066653 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.067063 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.067164 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:02.071183 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:02.071188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:02.071191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:02.071220 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:02.073717 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:02.074409 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:02.074414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:02.074416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:02.074532 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:02.075000 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:02.075005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:02.075010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:02.075260 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:02.075427 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:02.076053 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 03:00:02.076268 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":201,"threads_started":1}
I20260619 03:00:02.077493 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:02.057388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:53447)
0619 03:00:02.058026 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:02.058061 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:02.060556 (+ 2495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:02.061482 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:02.061494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:02.062335 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:00:02.063165 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:02.063192 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.064717 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.064722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:02.066467 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:02.066482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.067285 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.067294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:02.067625 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:02.068329 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:00:02.068363 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:00:02.070972 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:00:02.073848 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:02.073857 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:02.073871 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:02.074275 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:02.074676 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:02.074682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:02.074686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:02.074872 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:02.075386 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:00:02.075393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:00:02.075707 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 03:00:02.075965 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 03:00:02.076427 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":253,"threads_started":1}
I20260619 03:00:02.079000 6688 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36975
I20260619 03:00:02.079277 6688 heartbeater.cc:464] Registering TS with master...
I20260619 03:00:02.079931 6688 heartbeater.cc:511] Master 127.22.230.62:36975 requested a full tablet report, sending...
I20260619 03:00:02.082132 6499 ts_manager.cc:194] Registered new tserver with Master: ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605)
I20260619 03:00:02.084328 6499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:53447
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 03:00:02.117254 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36975
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:00:02.418915 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:02.419139 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:02.419219 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:02.425608 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:02.425693 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:02.425787 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:00:02.434789 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:02.435818 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:02.436910 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:02.449823 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:02.449857 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:02.449784 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:02.450161 6694 server_base.cc:1034] running on GCE node
I20260619 03:00:02.451247 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:02.452119 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:02.453332 6694 hybrid_clock.cc:648] HybridClock initialized: now 1781838002453279 us; error 43 us; skew 500 ppm
I20260619 03:00:02.453634 6694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:02 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:02.456707 6694 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 03:00:02.458120 6694 webserver.cc:460] Webserver started at http://127.22.230.2:37769/ using document root <none> and password file <none>
I20260619 03:00:02.458796 6694 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:02.458935 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:02.459203 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:00:02.461894 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "e8586a6b38a840f3a65a1cb7880911f8"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "2d2799abb5b1feb0c47f496ffc0ec853"
server_key_iv: "3b8ae8bc3f80a9348df1600baf0439ee"
server_key_version: "encryptionkey@0"
I20260619 03:00:02.462675 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "e8586a6b38a840f3a65a1cb7880911f8"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "2d2799abb5b1feb0c47f496ffc0ec853"
server_key_iv: "3b8ae8bc3f80a9348df1600baf0439ee"
server_key_version: "encryptionkey@0"
I20260619 03:00:02.467352 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260619 03:00:02.470459 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.471668 6694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:00:02.471838 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "e8586a6b38a840f3a65a1cb7880911f8"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "2d2799abb5b1feb0c47f496ffc0ec853"
server_key_iv: "3b8ae8bc3f80a9348df1600baf0439ee"
server_key_version: "encryptionkey@0"
I20260619 03:00:02.472020 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:02.496129 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:02.500174 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:02.500458 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:02.501351 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:02.502550 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:02.502717 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.502866 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:02.502935 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.533901 6694 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:46187
I20260619 03:00:02.533982 6821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:46187 every 8 connection(s)
I20260619 03:00:02.535120 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:00:02.535737 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 6694
I20260619 03:00:02.535998 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:00:02.536577 23448 external_mini_cluster.cc:1428] Setting key 070db3819f9bd49aee556345d624e279
Jun 19 03:00:02 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:02.554152 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:02.537747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:38693 (local address 127.22.230.62:36975)
0619 03:00:02.537944 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:02.537962 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:02.539180 (+ 1218us) server_negotiation.cc:407] Connection header received
0619 03:00:02.540946 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:02.540955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:02.541043 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:02.541148 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:02.542458 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.543523 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.544861 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.545521 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.545649 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:02.548791 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:02.548798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:02.548806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:02.548847 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:02.551389 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:02.552041 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:02.552047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:02.552049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:02.552177 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:02.552587 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:02.552593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:02.552596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:02.552859 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:02.553016 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:02.553623 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 03:00:02.553858 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 03:00:02.555032 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:02.538241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:38693)
0619 03:00:02.539008 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:02.539051 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:02.540684 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:02.541320 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:02.541327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:02.541799 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:00:02.542302 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:02.542317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.543655 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.543659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:02.544718 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:02.544729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:02.545706 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:02.545710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:02.545872 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:02.546464 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:00:02.546482 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:00:02.548600 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:00:02.551527 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:02.551535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:02.551549 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:02.551926 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:02.552281 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:02.552286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:02.552290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:02.552461 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:02.552997 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:00:02.553004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:00:02.553285 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 03:00:02.553537 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 03:00:02.553996 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":335,"threads_started":1}
I20260619 03:00:02.556324 6822 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36975
I20260619 03:00:02.556579 6822 heartbeater.cc:464] Registering TS with master...
I20260619 03:00:02.557257 6822 heartbeater.cc:511] Master 127.22.230.62:36975 requested a full tablet report, sending...
I20260619 03:00:02.558555 6499 ts_manager.cc:194] Registered new tserver with Master: e8586a6b38a840f3a65a1cb7880911f8 (127.22.230.2:46187)
I20260619 03:00:02.559327 6499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:38693
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 03:00:02.587055 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36975
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:00:02.889925 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:02.890213 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:02.890381 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:02.897274 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:02.897403 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:02.897588 6827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:00:02.907021 6827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:02.908185 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:02.909446 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:02.921396 6835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:02.921537 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:02.921473 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:02.922465 6827 server_base.cc:1034] running on GCE node
I20260619 03:00:02.923056 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:02.923856 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:02.925071 6827 hybrid_clock.cc:648] HybridClock initialized: now 1781838002925031 us; error 52 us; skew 500 ppm
I20260619 03:00:02.925354 6827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:02 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:02.928593 6827 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 03:00:02.929978 6827 webserver.cc:460] Webserver started at http://127.22.230.3:35297/ using document root <none> and password file <none>
I20260619 03:00:02.930675 6827 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:02.930817 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:02.931113 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:00:02.933923 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "d25c98e121b1406c9d03c65d13343412"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "826100882d2a126856b6c01e88198fdf"
server_key_iv: "98974a79a7feff3dc342db1487ba5472"
server_key_version: "encryptionkey@0"
I20260619 03:00:02.934744 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "d25c98e121b1406c9d03c65d13343412"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "826100882d2a126856b6c01e88198fdf"
server_key_iv: "98974a79a7feff3dc342db1487ba5472"
server_key_version: "encryptionkey@0"
I20260619 03:00:02.939806 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 03:00:02.943003 6842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.944244 6827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 03:00:02.944401 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "d25c98e121b1406c9d03c65d13343412"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "826100882d2a126856b6c01e88198fdf"
server_key_iv: "98974a79a7feff3dc342db1487ba5472"
server_key_version: "encryptionkey@0"
I20260619 03:00:02.944591 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:02.960047 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:02.963575 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:02.963848 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:02.964689 6827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:02.965924 6827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:02.965992 6827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.966157 6827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:02.966207 6827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:02.998538 6827 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:41255
I20260619 03:00:02.998644 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:41255 every 8 connection(s)
I20260619 03:00:02.999837 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:00:03.005828 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 6827
I20260619 03:00:03.006023 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:00:03.006475 23448 external_mini_cluster.cc:1428] Setting key a84b2aa2070038427c9cea34a233a5f5
Jun 19 03:00:03 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:03.018471 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.002108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:60227 (local address 127.22.230.62:36975)
0619 03:00:03.002303 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.002307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.003484 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 03:00:03.005349 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.005355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.005498 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.005615 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:03.007227 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.008022 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.009519 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.009923 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.010056 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.013144 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:03.013149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:03.013152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:03.013181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:03.015627 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.016297 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.016303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.016307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.016437 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.016900 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.016909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.016911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.017163 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:03.017319 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.017933 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.018137 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 03:00:03.019393 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.002498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:60227)
0619 03:00:03.003285 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:03.003374 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:03.005087 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:03.005808 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:03.005816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:03.006391 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:00:03.007050 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.007067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.008182 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.008186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.009359 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.009370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.010142 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.010150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.010378 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.011009 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:00:03.011032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:00:03.012948 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:00:03.015762 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:03.015771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:03.015785 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:03.016173 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:03.016595 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:03.016599 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:03.016603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:03.016778 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:03.017295 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:00:03.017302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:00:03.017593 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 03:00:03.017841 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 03:00:03.018301 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":296,"threads_started":1}
I20260619 03:00:03.020644 6955 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36975
I20260619 03:00:03.020936 6955 heartbeater.cc:464] Registering TS with master...
I20260619 03:00:03.021610 6955 heartbeater.cc:511] Master 127.22.230.62:36975 requested a full tablet report, sending...
I20260619 03:00:03.022938 6498 ts_manager.cc:194] Registered new tserver with Master: d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255)
I20260619 03:00:03.023727 6498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:60227
I20260619 03:00:03.033713 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:00:03.050833 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.038069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.22.230.62:36975)
0619 03:00:03.038237 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.038245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.038550 (+ 305us) server_negotiation.cc:407] Connection header received
0619 03:00:03.038760 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.038765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.038834 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.038930 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:03.040700 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.041433 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.042869 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.043267 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.043399 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.045757 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:03.045762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:03.045766 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:03.045801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:03.048249 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.048892 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.048903 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.048906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.049026 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.049379 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.049386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.049389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.049687 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:03.049943 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.050277 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.050485 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 03:00:03.054381 6498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:00:03.060938 6498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:00:03.087080 6688 heartbeater.cc:503] Master 127.22.230.62:36975 was elected leader, sending a full tablet report...
I20260619 03:00:03.094419 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.071610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.22.230.2:46187)
0619 03:00:03.072206 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.072217 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.085476 (+ 13259us) server_negotiation.cc:407] Connection header received
0619 03:00:03.086111 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.086122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.086442 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.086714 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:03.087454 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.088959 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.091785 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.093286 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.093413 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.093658 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.093925 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.094092 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":398,"threads_started":1}
I20260619 03:00:03.095345 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.071415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:46187 (local address 127.0.0.1:50398)
0619 03:00:03.085290 (+ 13875us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:03.085334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:03.085839 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:03.086760 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:03.086769 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:03.086811 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:03.087217 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.087234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.089151 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.089157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.091615 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.091634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.093465 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.093471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.093661 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.093717 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 03:00:03.094824 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0619 03:00:03.094989 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13582,"thread_start_us":16590,"threads_started":1}
I20260619 03:00:03.095984 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.071983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:41255 (local address 127.0.0.1:39056)
0619 03:00:03.087518 (+ 15535us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:03.087540 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:03.087687 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:03.088589 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:03.088594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:03.088670 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:03.089189 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.089205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.091429 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.091437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.093137 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.093149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.095210 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.095215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.095405 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.095447 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 03:00:03.095543 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 03:00:03.095666 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15377,"thread_start_us":12855,"threads_started":1}
I20260619 03:00:03.096306 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.071897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.22.230.1:40605)
0619 03:00:03.081790 (+ 9893us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.081801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.088728 (+ 6927us) server_negotiation.cc:407] Connection header received
0619 03:00:03.089048 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.089056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.089272 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.089432 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:03.090205 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.091245 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.093750 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.094985 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.095112 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.095329 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.095788 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.095934 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9689,"thread_start_us":9505,"threads_started":1}
I20260619 03:00:03.096293 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.072310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40605 (local address 127.0.0.1:46370)
0619 03:00:03.088633 (+ 16323us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:03.088649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:03.088817 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:03.089715 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:03.089725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:03.089744 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:03.090049 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.090056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.091400 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.091406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.093580 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.093594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.095473 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.095479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.095618 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.095640 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 03:00:03.095737 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 03:00:03.095871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15819,"mutex_wait_us":47,"thread_start_us":15276,"threads_started":1}
I20260619 03:00:03.098943 6757 tablet_service.cc:1467] Processing CreateTablet for tablet 52f63caf555045fdb4c63ba151ceb2fd (DEFAULT_TABLE table=test-table [id=8d5960eb397f4a6b8abe5dd7bd579596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:00:03.099505 6623 tablet_service.cc:1467] Processing CreateTablet for tablet 52f63caf555045fdb4c63ba151ceb2fd (DEFAULT_TABLE table=test-table [id=8d5960eb397f4a6b8abe5dd7bd579596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:00:03.100101 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f63caf555045fdb4c63ba151ceb2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:03.100651 6623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f63caf555045fdb4c63ba151ceb2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:03.101969 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.072310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.22.230.3:41255)
0619 03:00:03.084676 (+ 12366us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.084687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.087863 (+ 3176us) server_negotiation.cc:407] Connection header received
0619 03:00:03.087970 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.087977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.088179 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.088340 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:03.089430 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.091258 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.093293 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.094764 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.094879 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.095079 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.096240 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.096403 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12146,"thread_start_us":13184,"threads_started":1}
I20260619 03:00:03.105371 6890 tablet_service.cc:1467] Processing CreateTablet for tablet 52f63caf555045fdb4c63ba151ceb2fd (DEFAULT_TABLE table=test-table [id=8d5960eb397f4a6b8abe5dd7bd579596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:00:03.106709 6890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f63caf555045fdb4c63ba151ceb2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:03.111141 6973 tablet_bootstrap.cc:492] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Bootstrap starting.
I20260619 03:00:03.113947 6973 tablet_bootstrap.cc:654] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:03.114650 6974 tablet_bootstrap.cc:492] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Bootstrap starting.
I20260619 03:00:03.115306 6973 log.cc:824] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:03.118289 6973 tablet_bootstrap.cc:492] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: No bootstrap required, opened a new log
I20260619 03:00:03.118544 6974 tablet_bootstrap.cc:654] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:03.118584 6973 ts_tablet_manager.cc:1397] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 03:00:03.119786 6974 log.cc:824] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:03.120143 6976 tablet_bootstrap.cc:492] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Bootstrap starting.
I20260619 03:00:03.122996 6976 tablet_bootstrap.cc:654] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:03.123826 6976 log.cc:824] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:03.126076 6976 tablet_bootstrap.cc:492] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: No bootstrap required, opened a new log
I20260619 03:00:03.126312 6976 ts_tablet_manager.cc:1397] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 03:00:03.128043 6973 raft_consensus.cc:348] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.128310 6974 tablet_bootstrap.cc:492] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: No bootstrap required, opened a new log
I20260619 03:00:03.128360 6973 raft_consensus.cc:374] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:03.128476 6973 raft_consensus.cc:729] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2ab7a3cd5e43299ce351e30f8a1cb0, State: Initialized, Role: FOLLOWER
I20260619 03:00:03.128613 6974 ts_tablet_manager.cc:1397] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260619 03:00:03.129050 6973 consensus_queue.cc:260] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.130673 6973 ts_tablet_manager.cc:1428] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 03:00:03.136351 6976 raft_consensus.cc:348] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.136725 6976 raft_consensus.cc:374] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:03.136822 6976 raft_consensus.cc:729] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25c98e121b1406c9d03c65d13343412, State: Initialized, Role: FOLLOWER
I20260619 03:00:03.137372 6976 consensus_queue.cc:260] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.138281 6955 heartbeater.cc:503] Master 127.22.230.62:36975 was elected leader, sending a full tablet report...
I20260619 03:00:03.138926 6976 ts_tablet_manager.cc:1428] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260619 03:00:03.138833 6974 raft_consensus.cc:348] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.139271 6974 raft_consensus.cc:374] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:03.139427 6974 raft_consensus.cc:729] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8586a6b38a840f3a65a1cb7880911f8, State: Initialized, Role: FOLLOWER
I20260619 03:00:03.140089 6974 consensus_queue.cc:260] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.141121 6822 heartbeater.cc:503] Master 127.22.230.62:36975 was elected leader, sending a full tablet report...
I20260619 03:00:03.141741 6974 ts_tablet_manager.cc:1428] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260619 03:00:03.251387 6956 tablet.cc:2369] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:00:03.287159 6823 tablet.cc:2369] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:00:03.306807 6689 tablet.cc:2369] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:00:03.375379 6981 raft_consensus.cc:482] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:00:03.375689 6981 raft_consensus.cc:504] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.377223 6981 leader_election.cc:288] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605), d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255)
I20260619 03:00:03.383963 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.377769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:50555 (local address 127.22.230.1:40605)
0619 03:00:03.377993 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.378001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.378786 (+ 785us) server_negotiation.cc:407] Connection header received
0619 03:00:03.378938 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.378943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.379015 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.379101 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:03.379742 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.380595 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.382325 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.383089 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.383273 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.383349 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.383605 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.383720 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 03:00:03.383985 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.377617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:33737 (local address 127.22.230.3:41255)
0619 03:00:03.377825 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.377834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.378931 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 03:00:03.379133 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.379137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.379208 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.379302 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:03.379914 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.380593 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.382360 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.383094 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.383255 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.383332 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.383657 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.383760 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 03:00:03.384093 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.377616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40605 (local address 127.22.230.2:50555)
0619 03:00:03.378684 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:03.378699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:03.378809 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:03.379141 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:03.379146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:03.379168 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:03.379568 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.379578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.380752 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.380756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.382183 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.382194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.383305 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.383312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.383457 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.383487 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 03:00:03.383555 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 03:00:03.383679 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":31,"thread_start_us":625,"threads_started":1}
I20260619 03:00:03.384120 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.377696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:41255 (local address 127.22.230.2:33737)
0619 03:00:03.378836 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:03.378853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:03.378982 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:03.379333 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:03.379339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:03.379358 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:03.379750 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.379761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.380721 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.380728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.382194 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:03.382205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.383346 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.383352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:03.383493 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.383513 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 03:00:03.383614 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 03:00:03.383720 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":69,"thread_start_us":782,"threads_started":1}
I20260619 03:00:03.385133 6910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f63caf555045fdb4c63ba151ceb2fd" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25c98e121b1406c9d03c65d13343412" is_pre_election: true
I20260619 03:00:03.385133 6643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f63caf555045fdb4c63ba151ceb2fd" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" is_pre_election: true
I20260619 03:00:03.385658 6643 raft_consensus.cc:2454] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 0.
I20260619 03:00:03.385659 6910 raft_consensus.cc:2454] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 0.
I20260619 03:00:03.386341 6711 leader_election.cc:302] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca2ab7a3cd5e43299ce351e30f8a1cb0, e8586a6b38a840f3a65a1cb7880911f8; no voters:
I20260619 03:00:03.386806 6981 raft_consensus.cc:2781] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:00:03.386957 6981 raft_consensus.cc:482] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:00:03.387065 6981 raft_consensus.cc:3037] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:03.389293 6981 raft_consensus.cc:504] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.389904 6981 leader_election.cc:288] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 election: Requested vote from peers ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605), d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255)
I20260619 03:00:03.390211 6643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f63caf555045fdb4c63ba151ceb2fd" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
I20260619 03:00:03.390347 6910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f63caf555045fdb4c63ba151ceb2fd" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25c98e121b1406c9d03c65d13343412"
I20260619 03:00:03.390393 6643 raft_consensus.cc:3037] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:03.390545 6910 raft_consensus.cc:3037] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:03.392429 6643 raft_consensus.cc:2454] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 1.
I20260619 03:00:03.392640 6910 raft_consensus.cc:2454] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 1.
I20260619 03:00:03.392760 6711 leader_election.cc:302] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca2ab7a3cd5e43299ce351e30f8a1cb0, e8586a6b38a840f3a65a1cb7880911f8; no voters:
I20260619 03:00:03.393049 6981 raft_consensus.cc:2781] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:00:03.393523 6981 raft_consensus.cc:686] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 1 LEADER]: Becoming Leader. State: Replica: e8586a6b38a840f3a65a1cb7880911f8, State: Running, Role: LEADER
I20260619 03:00:03.393947 6981 consensus_queue.cc:237] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:03.398226 6498 catalog_manager.cc:5495] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8586a6b38a840f3a65a1cb7880911f8 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "e8586a6b38a840f3a65a1cb7880911f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20260619 03:00:03.443681 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.437905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.22.230.2:46187)
0619 03:00:03.438075 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.438082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.438120 (+ 38us) server_negotiation.cc:407] Connection header received
0619 03:00:03.438366 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.438369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.438443 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.438602 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:00:03.439012 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.440092 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.441495 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.441885 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.442023 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.442486 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:00:03.443092 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:00:03.443236 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.443311 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.443396 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 03:00:03.451411 6643 raft_consensus.cc:1261] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Refusing update from remote peer e8586a6b38a840f3a65a1cb7880911f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:00:03.451443 6910 raft_consensus.cc:1261] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Refusing update from remote peer e8586a6b38a840f3a65a1cb7880911f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:00:03.452368 6981 consensus_queue.cc:1035] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:00:03.452668 6984 consensus_queue.cc:1035] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:00:03.467628 6987 mvcc.cc:205] Tried to move back new op lower bound from 7298408462127624192 to 7298408461905047552. Current Snapshot: MvccSnapshot[applied={T|T < 7298408462127624192 or (T in {7298408462127624192})}]
I20260619 03:00:03.480160 6498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:00:03.480434 6498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:00:03.484268 6498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Sending DeleteTablet for 3 replicas of tablet 52f63caf555045fdb4c63ba151ceb2fd
I20260619 03:00:03.485237 6757 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f63caf555045fdb4c63ba151ceb2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:00:03 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50398
I20260619 03:00:03.485419 6890 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f63caf555045fdb4c63ba151ceb2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:00:03 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:39056
I20260619 03:00:03.485419 6623 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f63caf555045fdb4c63ba151ceb2fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:00:03 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:46370
I20260619 03:00:03.486328 6997 tablet_replica.cc:330] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: stopping tablet replica
I20260619 03:00:03.486919 6997 raft_consensus.cc:2229] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:00:03.487057 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 6560
I20260619 03:00:03.487612 6997 raft_consensus.cc:2258] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:00:03.490031 6997 ts_tablet_manager.cc:1905] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:00:03.498054 6997 ts_tablet_manager.cc:1918] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:00:03.498250 6997 log.cc:1196] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/52f63caf555045fdb4c63ba151ceb2fd
I20260619 03:00:03.498799 6997 ts_tablet_manager.cc:1939] T 52f63caf555045fdb4c63ba151ceb2fd P ca2ab7a3cd5e43299ce351e30f8a1cb0: Deleting consensus metadata
I20260619 03:00:03.500563 6484 catalog_manager.cc:4841] TS ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605): tablet 52f63caf555045fdb4c63ba151ceb2fd (table test-table [id=8d5960eb397f4a6b8abe5dd7bd579596]) successfully deleted
I20260619 03:00:03.506266 6996 tablet_replica.cc:330] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: stopping tablet replica
I20260619 03:00:03.506714 6996 raft_consensus.cc:2229] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:00:03.507215 6996 raft_consensus.cc:2258] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:00:03.508773 6996 ts_tablet_manager.cc:1905] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:00:03.512063 6998 tablet_replica.cc:330] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: stopping tablet replica
I20260619 03:00:03.512450 6998 raft_consensus.cc:2229] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:00:03.512944 6998 raft_consensus.cc:2258] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:00:03.515782 6996 ts_tablet_manager.cc:1918] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:00:03.516021 6996 log.cc:1196] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/52f63caf555045fdb4c63ba151ceb2fd
Jun 19 03:00:03 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 03:00:03.516615 6996 ts_tablet_manager.cc:1939] T 52f63caf555045fdb4c63ba151ceb2fd P e8586a6b38a840f3a65a1cb7880911f8: Deleting consensus metadata
I20260619 03:00:03.517571 6998 ts_tablet_manager.cc:1905] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:00:03.518703 6485 catalog_manager.cc:4841] TS e8586a6b38a840f3a65a1cb7880911f8 (127.22.230.2:46187): tablet 52f63caf555045fdb4c63ba151ceb2fd (table test-table [id=8d5960eb397f4a6b8abe5dd7bd579596]) successfully deleted
I20260619 03:00:03.523305 6998 ts_tablet_manager.cc:1918] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:00:03.523471 6998 log.cc:1196] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/52f63caf555045fdb4c63ba151ceb2fd
I20260619 03:00:03.523881 6998 ts_tablet_manager.cc:1939] T 52f63caf555045fdb4c63ba151ceb2fd P d25c98e121b1406c9d03c65d13343412: Deleting consensus metadata
I20260619 03:00:03.524780 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.488469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.22.230.1:40605)
0619 03:00:03.488693 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.488702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.503325 (+ 14623us) server_negotiation.cc:407] Connection header received
0619 03:00:03.503520 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.503529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.503632 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.503775 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:03.505978 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.507483 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.510119 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.510761 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.510931 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.517633 (+ 6702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:03.517647 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:03.517665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:03.517713 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:03.521480 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.522279 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.522288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.522292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.522441 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.522903 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.522908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.522911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.523317 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:03.523596 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.524226 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.524434 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 03:00:03.525418 6683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:03.525530 6486 catalog_manager.cc:4841] TS d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255): tablet 52f63caf555045fdb4c63ba151ceb2fd (table test-table [id=8d5960eb397f4a6b8abe5dd7bd579596]) successfully deleted
I20260619 03:00:03.628242 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 6560
I20260619 03:00:03.647111 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 6694
Jun 19 03:00:03 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 03:00:03.664867 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.647914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.22.230.2:46187)
0619 03:00:03.648073 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.648080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.648122 (+ 42us) server_negotiation.cc:407] Connection header received
0619 03:00:03.648340 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.648351 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.648417 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.648517 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:03.649820 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.650713 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.652677 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.653096 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.653183 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.658189 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:03.658208 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:03.658238 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:03.658310 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:03.662080 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.662835 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.662844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.662848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.663016 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.663343 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.663348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.663351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.663780 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:03.664091 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.664260 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.664473 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 03:00:03.665632 6817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:03.777733 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 6694
I20260619 03:00:03.796702 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 6827
Jun 19 03:00:03 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 03:00:03.811949 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:03.797467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.22.230.3:41255)
0619 03:00:03.797658 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:03.797665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:03.797759 (+ 94us) server_negotiation.cc:407] Connection header received
0619 03:00:03.797965 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:03.797971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:03.798052 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:03.798166 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:03.799433 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.800154 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.801510 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:03.801897 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:03.802037 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:03.806288 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:03.806307 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:03.806321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:03.806353 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:03.809122 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.809793 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.809800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.809803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.809938 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:03.810296 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:03.810302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:03.810306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:03.810798 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:03.811100 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:03.811420 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 03:00:03.811627 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 03:00:03.812568 6950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:03.899399 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 6827
I20260619 03:00:03.916365 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 6467
I20260619 03:00:03.917474 6529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:04.025871 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 6467
I20260619 03:00:24.047955 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36975
--webserver_interface=127.22.230.62
--webserver_port=42781
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:36975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:00:24.367311 7004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:24.367627 7004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:24.367779 7004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:24.374644 7004 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 03:00:24.374766 7004 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 03:00:24.374861 7004 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 03:00:24.374958 7004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:00:24.375057 7004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:24.375182 7004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:24.375273 7004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:00:24.375377 7004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:00:24.384654 7004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:36975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=42781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:24.385921 7004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:24.387171 7004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:24.409605 7011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:24.409663 7010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:24.409695 7013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:24.410063 7004 server_base.cc:1034] running on GCE node
I20260619 03:00:24.411067 7004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:24.412526 7004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:24.413774 7004 hybrid_clock.cc:648] HybridClock initialized: now 1781838024413725 us; error 42 us; skew 500 ppm
I20260619 03:00:24.414135 7004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:24 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838024, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:24.420535 7004 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:00:24.422046 7004 webserver.cc:460] Webserver started at http://127.22.230.62:42781/ using document root <none> and password file <none>
I20260619 03:00:24.422920 7004 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:24.423081 7004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:24.428793 7004 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:00:24.431586 7020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:24.432888 7004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:00:24.433087 7004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "541d6c93a16b44feb124cc41a20a5571"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "28e77a22cdfad18a9eba9582e6e67faa"
server_key_iv: "5396185b3644eafa11bdc1cf099a5138"
server_key_version: "encryptionkey@0"
I20260619 03:00:24.434110 7004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:24.454787 7004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:24.458700 7004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:24.459010 7004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:24.477469 7004 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:36975
I20260619 03:00:24.477497 7071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:36975 every 8 connection(s)
I20260619 03:00:24.478966 7004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:00:24.484273 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7004
I20260619 03:00:24.485263 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:40605
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:36975
--webserver_port=44133
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:00:24.486480 7072 sys_catalog.cc:263] Verifying existing consensus state
I20260619 03:00:24.491662 7072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap starting.
I20260619 03:00:24.511842 7072 log.cc:824] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:24.523278 7072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 03:00:24.523787 7072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap complete.
I20260619 03:00:24.532696 7072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:24.533445 7072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541d6c93a16b44feb124cc41a20a5571, State: Initialized, Role: FOLLOWER
I20260619 03:00:24.534118 7072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:24.534358 7072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:00:24.534489 7072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:00:24.534722 7072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 1 FOLLOWER]: Advancing to term 2
I20260619 03:00:24.538808 7072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:24.539314 7072 leader_election.cc:302] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541d6c93a16b44feb124cc41a20a5571; no voters:
I20260619 03:00:24.539922 7072 leader_election.cc:288] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 03:00:24.540040 7076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 03:00:24.541206 7076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 LEADER]: Becoming Leader. State: Replica: 541d6c93a16b44feb124cc41a20a5571, State: Running, Role: LEADER
I20260619 03:00:24.541811 7076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:24.542107 7072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:00:24.544190 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541d6c93a16b44feb124cc41a20a5571. Latest consensus state: current_term: 2 leader_uuid: "541d6c93a16b44feb124cc41a20a5571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } } }
I20260619 03:00:24.544526 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: This master's current role is: LEADER
I20260619 03:00:24.544286 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "541d6c93a16b44feb124cc41a20a5571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } } }
I20260619 03:00:24.544694 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: This master's current role is: LEADER
I20260619 03:00:24.555581 7081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:00:24.561292 7081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:00:24.562294 7081 catalog_manager.cc:1259] Loaded cluster ID: 464a29180a7a44ceb9c8adf70f7bd065
I20260619 03:00:24.562398 7081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:00:24.565001 7081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:00:24.566062 7081 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Loaded TSK: 0
I20260619 03:00:24.588378 7081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Generated new TSK 1
I20260619 03:00:24.588677 7081 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 03:00:24.835213 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:24.835552 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:24.835759 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:24.843148 7074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:24.843237 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:24.843437 7074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:00:24.853097 7074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:40605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=44133
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:24.854274 7074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:24.855576 7074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:24.869159 7100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:24.869563 7099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:24.871182 7102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:24.874918 7074 server_base.cc:1034] running on GCE node
I20260619 03:00:24.875635 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:24.876747 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:24.877991 7074 hybrid_clock.cc:648] HybridClock initialized: now 1781838024877921 us; error 54 us; skew 500 ppm
I20260619 03:00:24.878351 7074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:24 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838024, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:24.882156 7074 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 03:00:24.883664 7074 webserver.cc:460] Webserver started at http://127.22.230.1:44133/ using document root <none> and password file <none>
I20260619 03:00:24.884335 7074 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:24.884473 7074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:24.889635 7074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 03:00:24.892364 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:24.893676 7074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 03:00:24.893850 7074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "37289167011ebd386fda7a0013f9885c"
server_key_iv: "7f1a75c4fd23364be69adc676918fd78"
server_key_version: "encryptionkey@0"
I20260619 03:00:24.894840 7074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:24.913563 7074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:24.917646 7074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:24.917910 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:24.918886 7074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:24.920270 7074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:24.920387 7074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:24.920559 7074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:24.920652 7074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:24.953233 7074 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40605
I20260619 03:00:24.953286 7221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40605 every 8 connection(s)
I20260619 03:00:24.954622 7074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:00:24.961571 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7074
I20260619 03:00:24.962364 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:46187
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:36975
--webserver_port=37769
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 03:00:24 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838024, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:24.978116 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:24.957195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:55435 (local address 127.22.230.62:36975)
0619 03:00:24.957735 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:24.957746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:24.958354 (+ 608us) server_negotiation.cc:407] Connection header received
0619 03:00:24.960351 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:24.960363 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:24.960654 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:24.960952 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:24.962939 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:24.964085 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:24.966449 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:24.967152 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:24.967274 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:24.971196 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:24.971203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:24.971215 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:24.971246 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:24.974042 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:24.974809 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:24.974814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:24.974819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:24.974958 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:24.975358 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:24.975361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:24.975363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:24.975755 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:24.975956 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:24.976632 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 03:00:24.976979 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260619 03:00:24.978116 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:24.957421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:55435)
0619 03:00:24.958175 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:24.958217 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:24.959905 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:24.961173 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:24.961186 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:24.961932 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:00:24.962748 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:24.962774 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:24.964262 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:24.964269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:24.966313 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:24.966329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:24.967385 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:24.967394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:24.967702 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:24.968409 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:00:24.968446 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:00:24.970938 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:00:24.974195 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:24.974204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:24.974218 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:24.974679 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:24.975065 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:24.975070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:24.975074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:24.975237 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:24.975888 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:00:24.975895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:00:24.976201 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 03:00:24.976440 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 03:00:24.976917 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":322,"threads_started":1}
I20260619 03:00:24.980715 7222 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36975
I20260619 03:00:24.981062 7222 heartbeater.cc:464] Registering TS with master...
I20260619 03:00:24.981992 7222 heartbeater.cc:511] Master 127.22.230.62:36975 requested a full tablet report, sending...
I20260619 03:00:24.984201 7037 ts_manager.cc:194] Registered new tserver with Master: ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605)
I20260619 03:00:24.985589 7037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:55435
W20260619 03:00:25.279161 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:25.279433 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:25.279618 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:25.286669 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:25.286827 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:25.287019 7226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:00:25.297734 7226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:46187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=37769
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:25.299100 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:25.300340 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:25.312599 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:25.312670 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:25.312641 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:25.313330 7226 server_base.cc:1034] running on GCE node
I20260619 03:00:25.313882 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:25.314813 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:25.316004 7226 hybrid_clock.cc:648] HybridClock initialized: now 1781838025315945 us; error 58 us; skew 500 ppm
I20260619 03:00:25.316293 7226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:25 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:25.319209 7226 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 03:00:25.320438 7226 webserver.cc:460] Webserver started at http://127.22.230.2:37769/ using document root <none> and password file <none>
I20260619 03:00:25.321031 7226 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:25.321136 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:25.325944 7226 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 03:00:25.328366 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:25.329574 7226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:00:25.329737 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "e8586a6b38a840f3a65a1cb7880911f8"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "2d2799abb5b1feb0c47f496ffc0ec853"
server_key_iv: "3b8ae8bc3f80a9348df1600baf0439ee"
server_key_version: "encryptionkey@0"
I20260619 03:00:25.330688 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:25.360364 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:25.364106 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:25.364358 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:25.365597 7226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:25.367038 7226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:25.367156 7226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:25.367286 7226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:25.367377 7226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:25.399821 7226 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:46187
I20260619 03:00:25.399899 7353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:46187 every 8 connection(s)
I20260619 03:00:25.401212 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:00:25.406855 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7226
I20260619 03:00:25.407657 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:41255
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:36975
--webserver_port=35297
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 03:00:25 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:25.422431 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.403581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:47681 (local address 127.22.230.62:36975)
0619 03:00:25.403780 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.403785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.404989 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 03:00:25.406981 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.406987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.407070 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.407151 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:25.409233 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.409906 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.412389 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.412879 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.412993 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.417010 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:25.417016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:25.417019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:25.417050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:25.419572 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.420267 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.420277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.420280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.420394 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.420817 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.420823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.420831 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.421078 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:25.421233 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.421867 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.422106 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 03:00:25.423368 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.404098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:47681)
0619 03:00:25.404805 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:25.404864 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:25.406571 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:25.407428 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:25.407438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:25.408234 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:00:25.409054 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.409083 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.410067 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.410074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.412225 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.412244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.413108 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.413118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.413444 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.414167 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:00:25.414199 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:00:25.416785 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:00:25.419713 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:25.419721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:25.419736 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:25.420137 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:25.420504 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:25.420509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:25.420513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:25.420688 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:25.421204 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:00:25.421211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:00:25.421518 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 03:00:25.421764 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 03:00:25.422245 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
I20260619 03:00:25.424809 7354 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36975
I20260619 03:00:25.425086 7354 heartbeater.cc:464] Registering TS with master...
I20260619 03:00:25.425769 7354 heartbeater.cc:511] Master 127.22.230.62:36975 requested a full tablet report, sending...
I20260619 03:00:25.427286 7037 ts_manager.cc:194] Registered new tserver with Master: e8586a6b38a840f3a65a1cb7880911f8 (127.22.230.2:46187)
I20260619 03:00:25.428588 7037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:47681
W20260619 03:00:25.723418 7357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:25.723711 7357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:25.723809 7357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:25.730933 7357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:25.731069 7357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:25.731190 7357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:00:25.740944 7357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:41255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=35297
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:25.742249 7357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:25.743566 7357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:25.756266 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:25.756306 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:25.756292 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:25.757244 7357 server_base.cc:1034] running on GCE node
I20260619 03:00:25.757844 7357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:25.758826 7357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:25.760016 7357 hybrid_clock.cc:648] HybridClock initialized: now 1781838025759957 us; error 51 us; skew 500 ppm
I20260619 03:00:25.760313 7357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:25 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:25.763370 7357 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 03:00:25.764798 7357 webserver.cc:460] Webserver started at http://127.22.230.3:35297/ using document root <none> and password file <none>
I20260619 03:00:25.765452 7357 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:25.765614 7357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:25.770783 7357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 03:00:25.773481 7372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:25.774933 7357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:00:25.775102 7357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "d25c98e121b1406c9d03c65d13343412"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "826100882d2a126856b6c01e88198fdf"
server_key_iv: "98974a79a7feff3dc342db1487ba5472"
server_key_version: "encryptionkey@0"
I20260619 03:00:25.776047 7357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:25.798036 7357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:25.801779 7357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:25.802103 7357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:25.803033 7357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:25.804291 7357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:25.804415 7357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:25.804545 7357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:25.804631 7357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:25.838063 7357 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:41255
I20260619 03:00:25.838107 7484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:41255 every 8 connection(s)
I20260619 03:00:25.839644 7357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:00:25.842936 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7357
Jun 19 03:00:25 dist-test-slave-2gms krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838025, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:00:25.858884 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.844230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.22.230.62:36975)
0619 03:00:25.844689 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.844694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.844712 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:00:25.844777 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.844780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.844841 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.844962 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:25.846498 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.847417 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.848839 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.849281 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.849417 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.852432 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:25.852438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:25.852443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:25.852481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:25.856055 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.856764 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.856770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.856773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.856936 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.857292 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.857298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.857302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.857650 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:25.857906 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.858238 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.858466 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260619 03:00:25.860620 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.842206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:48813 (local address 127.22.230.62:36975)
0619 03:00:25.842382 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.842387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.843650 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 03:00:25.845649 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.845654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.845734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.845846 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:25.847484 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.848204 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.849801 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.850404 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.850574 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.854196 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:25.854202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:25.854209 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:25.854267 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:25.857422 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.858095 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.858102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.858110 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.858225 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.858799 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.858803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.858805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.859126 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:25.859316 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.860031 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.860252 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 03:00:25.861624 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.842681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:48813)
0619 03:00:25.843478 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:25.843522 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:25.845400 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:25.846076 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:25.846083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:25.846639 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:00:25.847318 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.847334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.848355 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.848366 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.849647 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.849658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.850699 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.850707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.850969 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.851597 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:00:25.851622 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:00:25.853943 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:00:25.857553 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:25.857559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:25.857571 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:25.857971 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:25.858343 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:00:25.858349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:00:25.858352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:00:25.858524 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:00:25.859289 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:00:25.859297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:00:25.859670 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 03:00:25.859951 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 03:00:25.860479 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":317,"threads_started":1}
I20260619 03:00:25.862872 7485 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36975
I20260619 03:00:25.863163 7485 heartbeater.cc:464] Registering TS with master...
I20260619 03:00:25.863826 7485 heartbeater.cc:511] Master 127.22.230.62:36975 requested a full tablet report, sending...
I20260619 03:00:25.865141 7037 ts_manager.cc:194] Registered new tserver with Master: d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255)
I20260619 03:00:25.865896 7037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:48813
I20260619 03:00:25.873396 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:00:25.879694 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.874317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.22.230.62:36975)
0619 03:00:25.874482 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.874486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.874838 (+ 352us) server_negotiation.cc:407] Connection header received
0619 03:00:25.875001 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.875005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.875130 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.875243 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:00:25.875655 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.876365 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.877827 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.878204 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.878290 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.878795 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:00:25.878984 (+ 189us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 03:00:25.879387 (+ 403us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45986: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 03:00:25.879967 7224 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45986: token expired
W20260619 03:00:25.880239 6960 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.230.62:36975)
I20260619 03:00:25.892223 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.880644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.22.230.62:36975)
0619 03:00:25.880793 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.880797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.880845 (+ 48us) server_negotiation.cc:407] Connection header received
0619 03:00:25.880970 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.880973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.881019 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.881089 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:25.882440 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.883199 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.884498 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.884862 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.884941 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.887425 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:25.887429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:25.887432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:25.887460 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:25.890097 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.890753 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.890757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.890759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.890866 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:25.891159 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:25.891162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:25.891164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:25.891440 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:25.891609 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.891846 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.891993 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 03:00:25.895282 7037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:00:25.902325 7037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:00:25.925182 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.913814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40605 (local address 127.0.0.1:40644)
0619 03:00:25.914808 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:25.914856 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:25.915079 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:25.916996 (+ 1917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:25.917008 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:25.917051 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:25.917656 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.917671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.919651 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.919657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.922168 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.922182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.924197 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.924203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.924406 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.924468 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 03:00:25.924614 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 03:00:25.924792 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":492,"threads_started":1}
I20260619 03:00:25.925554 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.913949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:46187 (local address 127.0.0.1:37646)
0619 03:00:25.917569 (+ 3620us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:25.917600 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:25.917725 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:25.918658 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:25.918663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:25.918687 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:25.919099 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.919109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.920696 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.920702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.922881 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.922893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.924600 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.924608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.924758 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.924787 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 03:00:25.924872 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 03:00:25.925002 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"mutex_wait_us":24,"thread_start_us":1792,"threads_started":1}
I20260619 03:00:25.925669 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.914469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.22.230.2:46187)
0619 03:00:25.915332 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.915344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.918101 (+ 2757us) server_negotiation.cc:407] Connection header received
0619 03:00:25.918210 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.918219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.918464 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.918677 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:25.919334 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.920551 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.923063 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.924367 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.924508 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.924688 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.924930 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.925104 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":563,"threads_started":1}
I20260619 03:00:25.926028 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.913988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.22.230.1:40605)
0619 03:00:25.915658 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.915669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.915700 (+ 31us) server_negotiation.cc:407] Connection header received
0619 03:00:25.915812 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.915821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.916089 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.917320 (+ 1231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:25.917882 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.919479 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.922513 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.923985 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.924141 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.924346 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.925515 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.925703 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1525,"thread_start_us":1422,"threads_started":1}
I20260619 03:00:25.927016 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.916567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.22.230.3:41255)
0619 03:00:25.917216 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:25.917221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:25.918282 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 03:00:25.918413 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:25.918420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:25.919253 (+ 833us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:25.919437 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:25.920384 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.921571 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.923871 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.925569 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.925702 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.925931 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:25.926458 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 03:00:25.926646 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":921,"threads_started":1}
I20260619 03:00:25.927223 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:25.916181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:41255 (local address 127.0.0.1:33766)
0619 03:00:25.918155 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:25.918177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:25.918290 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:25.919736 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:25.919742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:25.919778 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:25.920200 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.920211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.921708 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.921714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.923707 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:25.923722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:25.926087 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:25.926093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:25.926257 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:25.926297 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 03:00:25.926388 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 03:00:25.926901 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1853,"thread_start_us":1705,"threads_started":1}
I20260619 03:00:25.929471 7157 tablet_service.cc:1467] Processing CreateTablet for tablet 213b25c845744b0b803f0eb980fe7c94 (DEFAULT_TABLE table=test-table [id=06d05d84540f4e50b2d3f62933ed063f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:00:25.929659 7289 tablet_service.cc:1467] Processing CreateTablet for tablet 213b25c845744b0b803f0eb980fe7c94 (DEFAULT_TABLE table=test-table [id=06d05d84540f4e50b2d3f62933ed063f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:00:25.930702 7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213b25c845744b0b803f0eb980fe7c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:25.930960 7289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213b25c845744b0b803f0eb980fe7c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:25.930738 7420 tablet_service.cc:1467] Processing CreateTablet for tablet 213b25c845744b0b803f0eb980fe7c94 (DEFAULT_TABLE table=test-table [id=06d05d84540f4e50b2d3f62933ed063f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:00:25.931972 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213b25c845744b0b803f0eb980fe7c94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:00:25.942886 7501 tablet_bootstrap.cc:492] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Bootstrap starting.
I20260619 03:00:25.945631 7502 tablet_bootstrap.cc:492] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Bootstrap starting.
I20260619 03:00:25.946908 7503 tablet_bootstrap.cc:492] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Bootstrap starting.
I20260619 03:00:25.947628 7501 tablet_bootstrap.cc:654] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:25.948849 7501 log.cc:824] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:25.949095 7502 tablet_bootstrap.cc:654] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:25.949767 7503 tablet_bootstrap.cc:654] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Neither blocks nor log segments found. Creating new log.
I20260619 03:00:25.950445 7502 log.cc:824] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:25.951171 7503 log.cc:824] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:25.952126 7501 tablet_bootstrap.cc:492] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: No bootstrap required, opened a new log
I20260619 03:00:25.952461 7501 ts_tablet_manager.cc:1397] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 03:00:25.953015 7502 tablet_bootstrap.cc:492] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: No bootstrap required, opened a new log
I20260619 03:00:25.953310 7502 ts_tablet_manager.cc:1397] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 03:00:25.955618 7503 tablet_bootstrap.cc:492] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: No bootstrap required, opened a new log
I20260619 03:00:25.955909 7503 ts_tablet_manager.cc:1397] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 03:00:25.963635 7502 raft_consensus.cc:348] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:25.964150 7502 raft_consensus.cc:374] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:25.964285 7502 raft_consensus.cc:729] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8586a6b38a840f3a65a1cb7880911f8, State: Initialized, Role: FOLLOWER
I20260619 03:00:25.965046 7502 consensus_queue.cc:260] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:25.966307 7503 raft_consensus.cc:348] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:25.966774 7503 raft_consensus.cc:374] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:25.966904 7503 raft_consensus.cc:729] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25c98e121b1406c9d03c65d13343412, State: Initialized, Role: FOLLOWER
I20260619 03:00:25.967314 7501 raft_consensus.cc:348] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:25.967564 7503 consensus_queue.cc:260] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:25.967826 7501 raft_consensus.cc:374] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:00:25.967993 7501 raft_consensus.cc:729] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2ab7a3cd5e43299ce351e30f8a1cb0, State: Initialized, Role: FOLLOWER
I20260619 03:00:25.968698 7501 consensus_queue.cc:260] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:25.970237 7485 heartbeater.cc:503] Master 127.22.230.62:36975 was elected leader, sending a full tablet report...
I20260619 03:00:25.970907 7503 ts_tablet_manager.cc:1428] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260619 03:00:25.972011 7354 heartbeater.cc:503] Master 127.22.230.62:36975 was elected leader, sending a full tablet report...
I20260619 03:00:25.972258 7222 heartbeater.cc:503] Master 127.22.230.62:36975 was elected leader, sending a full tablet report...
I20260619 03:00:25.972555 7502 ts_tablet_manager.cc:1428] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260619 03:00:25.972838 7501 ts_tablet_manager.cc:1428] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Time spent starting tablet: real 0.020s user 0.013s sys 0.003s
I20260619 03:00:26.008181 7507 raft_consensus.cc:482] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:00:26.008494 7507 raft_consensus.cc:504] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:26.010113 7507 leader_election.cc:288] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605), d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255)
I20260619 03:00:26.017153 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.010592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:53727 (local address 127.22.230.1:40605)
0619 03:00:26.010779 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:26.010783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:26.011590 (+ 807us) server_negotiation.cc:407] Connection header received
0619 03:00:26.011756 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:26.011761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:26.011836 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:26.011946 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:26.012714 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.013631 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.015486 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.016234 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.016407 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.016487 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:26.016771 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 03:00:26.016888 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 03:00:26.017278 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.010434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:40769 (local address 127.22.230.3:41255)
0619 03:00:26.010680 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:26.010687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:26.011545 (+ 858us) server_negotiation.cc:407] Connection header received
0619 03:00:26.011779 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:26.011784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:26.011850 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:26.011936 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:00:26.012763 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.013727 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.015551 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.016318 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.016519 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.016590 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:26.016873 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 03:00:26.016995 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 03:00:26.017329 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.010710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:40605 (local address 127.22.230.2:53727)
0619 03:00:26.011450 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:26.011464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:26.011596 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:26.011971 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:26.011976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:26.012017 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:26.012513 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:26.012527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.013759 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.013767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:26.015333 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:26.015344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.016460 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.016466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:26.016605 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.016630 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 03:00:26.016734 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 03:00:26.016853 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":44,"spinlock_wait_cycles":4992,"thread_start_us":444,"threads_started":1}
I20260619 03:00:26.017436 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.010675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:41255 (local address 127.22.230.2:40769)
0619 03:00:26.011424 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:26.011452 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 03:00:26.011651 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:00:26.011971 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:00:26.011976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:00:26.012010 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:00:26.012646 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:26.012659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.013870 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.013876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:26.015402 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:00:26.015415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.016552 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.016558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:00:26.016713 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.016735 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 03:00:26.016839 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 03:00:26.016954 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":62,"spinlock_wait_cycles":37376,"thread_start_us":375,"threads_started":1}
I20260619 03:00:26.018311 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213b25c845744b0b803f0eb980fe7c94" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25c98e121b1406c9d03c65d13343412" is_pre_election: true
I20260619 03:00:26.018399 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213b25c845744b0b803f0eb980fe7c94" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" is_pre_election: true
I20260619 03:00:26.018847 7440 raft_consensus.cc:2454] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 0.
I20260619 03:00:26.018926 7177 raft_consensus.cc:2454] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 0.
I20260619 03:00:26.019465 7245 leader_election.cc:302] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25c98e121b1406c9d03c65d13343412, e8586a6b38a840f3a65a1cb7880911f8; no voters:
I20260619 03:00:26.019832 7507 raft_consensus.cc:2781] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:00:26.019977 7507 raft_consensus.cc:482] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:00:26.020080 7507 raft_consensus.cc:3037] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:26.022125 7507 raft_consensus.cc:504] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:26.022719 7507 leader_election.cc:288] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 election: Requested vote from peers ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605), d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255)
I20260619 03:00:26.023027 7177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213b25c845744b0b803f0eb980fe7c94" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
I20260619 03:00:26.023205 7177 raft_consensus.cc:3037] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:26.023185 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213b25c845744b0b803f0eb980fe7c94" candidate_uuid: "e8586a6b38a840f3a65a1cb7880911f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25c98e121b1406c9d03c65d13343412"
I20260619 03:00:26.023378 7440 raft_consensus.cc:3037] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:00:26.025367 7177 raft_consensus.cc:2454] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 1.
I20260619 03:00:26.025550 7440 raft_consensus.cc:2454] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8586a6b38a840f3a65a1cb7880911f8 in term 1.
I20260619 03:00:26.025741 7243 leader_election.cc:302] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca2ab7a3cd5e43299ce351e30f8a1cb0, e8586a6b38a840f3a65a1cb7880911f8; no voters:
I20260619 03:00:26.026012 7507 raft_consensus.cc:2781] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:00:26.026439 7507 raft_consensus.cc:686] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 1 LEADER]: Becoming Leader. State: Replica: e8586a6b38a840f3a65a1cb7880911f8, State: Running, Role: LEADER
I20260619 03:00:26.026966 7507 consensus_queue.cc:237] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } }
I20260619 03:00:26.031064 7036 catalog_manager.cc:5495] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8586a6b38a840f3a65a1cb7880911f8 (127.22.230.2). New cstate: current_term: 1 leader_uuid: "e8586a6b38a840f3a65a1cb7880911f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8586a6b38a840f3a65a1cb7880911f8" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 46187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 } health_report { overall_health: UNKNOWN } } }
I20260619 03:00:26.047119 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.042467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.22.230.2:46187)
0619 03:00:26.042665 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:26.042669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:26.042685 (+ 16us) server_negotiation.cc:407] Connection header received
0619 03:00:26.042851 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:26.042854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:26.042920 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:26.043026 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:00:26.043370 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.044030 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.045291 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.045662 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.045743 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.046136 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:00:26.046539 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:00:26.046733 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:26.046793 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 03:00:26.046849 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 03:00:26.054080 7440 raft_consensus.cc:1261] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Refusing update from remote peer e8586a6b38a840f3a65a1cb7880911f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:00:26.054103 7177 raft_consensus.cc:1261] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Refusing update from remote peer e8586a6b38a840f3a65a1cb7880911f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:00:26.055042 7512 consensus_queue.cc:1035] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:00:26.055385 7507 consensus_queue.cc:1035] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25c98e121b1406c9d03c65d13343412" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:00:26.093469 7037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:00:26.093765 7037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:00:26.096918 7037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Sending DeleteTablet for 3 replicas of tablet 213b25c845744b0b803f0eb980fe7c94
I20260619 03:00:26.097980 7157 tablet_service.cc:1514] Processing DeleteTablet for tablet 213b25c845744b0b803f0eb980fe7c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:00:26 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:40644
I20260619 03:00:26.098326 7420 tablet_service.cc:1514] Processing DeleteTablet for tablet 213b25c845744b0b803f0eb980fe7c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:00:26 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:33766
I20260619 03:00:26.098488 7289 tablet_service.cc:1514] Processing DeleteTablet for tablet 213b25c845744b0b803f0eb980fe7c94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:00:26 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:37646
I20260619 03:00:26.099576 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7074
I20260619 03:00:26.109239 7525 tablet_replica.cc:330] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: stopping tablet replica
I20260619 03:00:26.109757 7525 raft_consensus.cc:2229] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:00:26.110321 7525 raft_consensus.cc:2258] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:00:26.112442 7525 ts_tablet_manager.cc:1905] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:00:26.119344 7525 ts_tablet_manager.cc:1918] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:00:26.119567 7525 log.cc:1196] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/213b25c845744b0b803f0eb980fe7c94
I20260619 03:00:26.120258 7525 ts_tablet_manager.cc:1939] T 213b25c845744b0b803f0eb980fe7c94 P ca2ab7a3cd5e43299ce351e30f8a1cb0: Deleting consensus metadata
I20260619 03:00:26.122061 7022 catalog_manager.cc:4841] TS ca2ab7a3cd5e43299ce351e30f8a1cb0 (127.22.230.1:40605): tablet 213b25c845744b0b803f0eb980fe7c94 (table test-table [id=06d05d84540f4e50b2d3f62933ed063f]) successfully deleted
I20260619 03:00:26.130683 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.101402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.22.230.1:40605)
0619 03:00:26.101620 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:26.101628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:26.101651 (+ 23us) server_negotiation.cc:407] Connection header received
0619 03:00:26.101719 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:26.101726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:26.101835 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:26.101951 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:26.104462 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.105945 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.108472 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.109454 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.109600 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.119104 (+ 9504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:26.119123 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:26.119145 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:26.119196 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:26.124152 (+ 4956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:26.125319 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:26.125332 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:26.125338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:26.125524 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:26.127045 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:26.127053 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:26.127059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:26.127649 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:26.128017 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:26.129864 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0619 03:00:26.130149 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 03:00:26.131831 7217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:26.133919 7527 tablet_replica.cc:330] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: stopping tablet replica
I20260619 03:00:26.134356 7527 raft_consensus.cc:2229] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:00:26.135251 7527 raft_consensus.cc:2258] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:00:26.136382 7526 tablet_replica.cc:330] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: stopping tablet replica
I20260619 03:00:26.136897 7526 raft_consensus.cc:2229] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:00:26.137423 7526 raft_consensus.cc:2258] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:00:26.137661 7527 ts_tablet_manager.cc:1905] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:00:26.139842 7526 ts_tablet_manager.cc:1905] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:00:26.145020 7527 ts_tablet_manager.cc:1918] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:00:26.145221 7527 log.cc:1196] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/213b25c845744b0b803f0eb980fe7c94
I20260619 03:00:26.145778 7527 ts_tablet_manager.cc:1939] T 213b25c845744b0b803f0eb980fe7c94 P e8586a6b38a840f3a65a1cb7880911f8: Deleting consensus metadata
I20260619 03:00:26.147543 7023 catalog_manager.cc:4841] TS e8586a6b38a840f3a65a1cb7880911f8 (127.22.230.2:46187): tablet 213b25c845744b0b803f0eb980fe7c94 (table test-table [id=06d05d84540f4e50b2d3f62933ed063f]) successfully deleted
I20260619 03:00:26.148568 7526 ts_tablet_manager.cc:1918] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:00:26.148725 7526 log.cc:1196] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/213b25c845744b0b803f0eb980fe7c94
I20260619 03:00:26.149148 7526 ts_tablet_manager.cc:1939] T 213b25c845744b0b803f0eb980fe7c94 P d25c98e121b1406c9d03c65d13343412: Deleting consensus metadata
I20260619 03:00:26.150514 7024 catalog_manager.cc:4841] TS d25c98e121b1406c9d03c65d13343412 (127.22.230.3:41255): tablet 213b25c845744b0b803f0eb980fe7c94 (table test-table [id=06d05d84540f4e50b2d3f62933ed063f]) successfully deleted
I20260619 03:00:26.253808 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7074
I20260619 03:00:26.271772 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7226
I20260619 03:00:26.286309 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.272526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.22.230.2:46187)
0619 03:00:26.272716 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:26.272726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:26.272749 (+ 23us) server_negotiation.cc:407] Connection header received
0619 03:00:26.272970 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:26.272974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:26.273034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:26.273130 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:26.274411 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.275269 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.276572 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.277033 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.277158 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.279889 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:26.279902 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:26.279933 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:26.279970 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:26.283689 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:26.284267 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:26.284276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:26.284280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:26.284475 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:26.284812 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:26.284817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:26.284820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:26.285258 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:26.285586 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:26.285719 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 03:00:26.285931 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:00:26.287277 7349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:26.396175 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7226
I20260619 03:00:26.414215 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7357
I20260619 03:00:26.426402 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:26.414931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.22.230.3:41255)
0619 03:00:26.415101 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 03:00:26.415105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:00:26.415121 (+ 16us) server_negotiation.cc:407] Connection header received
0619 03:00:26.415218 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:00:26.415221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:00:26.415291 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:00:26.415373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:00:26.416599 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.417297 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.418433 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:00:26.418828 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:00:26.418912 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:00:26.421153 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:00:26.421164 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:00:26.421178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:00:26.421207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:00:26.424213 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:26.424800 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:26.424806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:26.424807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:26.424930 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:00:26.425211 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:00:26.425213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:00:26.425215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:00:26.425515 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:00:26.425710 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 03:00:26.426049 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 03:00:26.426184 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 03:00:26.427256 7480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:26.512465 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7357
I20260619 03:00:26.528825 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7004
I20260619 03:00:26.529906 7067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:00:26.638746 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7004
I20260619 03:00:46.660745 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36975
--webserver_interface=127.22.230.62
--webserver_port=42781
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:36975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:00:46.994786 7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:46.995075 7532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:46.995244 7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:47.004436 7532 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 03:00:47.004547 7532 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 03:00:47.004607 7532 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 03:00:47.004683 7532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:00:47.004787 7532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:47.004885 7532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:47.004943 7532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:00:47.005033 7532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:00:47.014642 7532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:36975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=42781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:47.015880 7532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:47.017168 7532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:47.028839 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:47.028900 7537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:47.028923 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:47.029408 7532 server_base.cc:1034] running on GCE node
I20260619 03:00:47.030174 7532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:47.031481 7532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:47.032718 7532 hybrid_clock.cc:648] HybridClock initialized: now 1781838047032646 us; error 54 us; skew 500 ppm
I20260619 03:00:47.033008 7532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:47 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838047, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:47.036283 7532 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:00:47.037775 7532 webserver.cc:460] Webserver started at http://127.22.230.62:42781/ using document root <none> and password file <none>
I20260619 03:00:47.038383 7532 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:47.038470 7532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:47.043478 7532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 03:00:47.046315 7547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.047565 7532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:00:47.047719 7532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "541d6c93a16b44feb124cc41a20a5571"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "28e77a22cdfad18a9eba9582e6e67faa"
server_key_iv: "5396185b3644eafa11bdc1cf099a5138"
server_key_version: "encryptionkey@0"
I20260619 03:00:47.048611 7532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:47.069515 7532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:47.072929 7532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:47.073196 7532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:47.090611 7532 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:36975
I20260619 03:00:47.090682 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:36975 every 8 connection(s)
I20260619 03:00:47.091878 7532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:00:47.094977 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7532
I20260619 03:00:47.095716 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:40605
--local_ip_for_outbound_sockets=127.22.230.1
--tserver_master_addrs=127.22.230.62:36975
--webserver_port=44133
--webserver_interface=127.22.230.1
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:00:47.098150 7599 sys_catalog.cc:263] Verifying existing consensus state
I20260619 03:00:47.101007 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap starting.
I20260619 03:00:47.120280 7599 log.cc:824] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Log is configured to *not* fsync() on all Append() calls
I20260619 03:00:47.135174 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 03:00:47.135727 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Bootstrap complete.
I20260619 03:00:47.144459 7599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:47.145231 7599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541d6c93a16b44feb124cc41a20a5571, State: Initialized, Role: FOLLOWER
I20260619 03:00:47.145824 7599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:47.146050 7599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:00:47.146183 7599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:00:47.146373 7599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 2 FOLLOWER]: Advancing to term 3
I20260619 03:00:47.149943 7599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:47.150401 7599 leader_election.cc:302] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541d6c93a16b44feb124cc41a20a5571; no voters:
I20260619 03:00:47.151070 7599 leader_election.cc:288] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 03:00:47.151188 7603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 03:00:47.152323 7603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [term 3 LEADER]: Becoming Leader. State: Replica: 541d6c93a16b44feb124cc41a20a5571, State: Running, Role: LEADER
I20260619 03:00:47.152901 7603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } }
I20260619 03:00:47.153043 7599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:00:47.155452 7604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "541d6c93a16b44feb124cc41a20a5571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } } }
I20260619 03:00:47.155836 7604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: This master's current role is: LEADER
I20260619 03:00:47.159592 7605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541d6c93a16b44feb124cc41a20a5571. Latest consensus state: current_term: 3 leader_uuid: "541d6c93a16b44feb124cc41a20a5571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541d6c93a16b44feb124cc41a20a5571" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36975 } } }
I20260619 03:00:47.160014 7605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571 [sys.catalog]: This master's current role is: LEADER
I20260619 03:00:47.161366 7609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:00:47.171223 7609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:00:47.173076 7609 catalog_manager.cc:1259] Loaded cluster ID: 464a29180a7a44ceb9c8adf70f7bd065
I20260619 03:00:47.173254 7609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:00:47.178412 7609 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:00:47.180460 7609 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Loaded TSK: 0
I20260619 03:00:47.180693 7609 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 541d6c93a16b44feb124cc41a20a5571: Loaded TSK: 1
F20260619 03:00:47.182067 7609 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f11f80269ad google::LogMessage::Fail() at ??:0
@ 0x7f11f802988a google::LogMessage::SendToLog() at ??:0
@ 0x7f11f80264de google::LogMessage::Flush() at ??:0
@ 0x7f11f8027a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f11fbd8a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f11fbd7973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f120da551c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f120da54783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f120dae02a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f120dae00c4 std::_Function_handler<>::_M_invoke() at ??:0
I20260619 03:00:47.224164 7623 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f120db2a5e9 std::function<>::operator()() at ??:0
@ 0x7f120da58cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f120da56c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f120dade610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f120dade439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f11fac72345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f11fac7591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f11fac75799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f11fac50eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f11fdf2a6db start_thread at ??:0
@ 0x7f11f6ec061f clone at ??:0
W20260619 03:00:47.457593 7601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:47.457860 7601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:47.458021 7601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:47.464884 7601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:47.465026 7601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:47.465173 7601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:00:47.474092 7601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:40605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=44133
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:47.475247 7601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:47.476377 7601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:47.488639 7647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:47.488581 7646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:47.488667 7649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:47.489203 7601 server_base.cc:1034] running on GCE node
I20260619 03:00:47.489751 7601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:47.490537 7601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:47.491799 7601 hybrid_clock.cc:648] HybridClock initialized: now 1781838047491757 us; error 34 us; skew 500 ppm
I20260619 03:00:47.492082 7601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:47 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838047, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:47.494966 7601 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 03:00:47.496155 7601 webserver.cc:460] Webserver started at http://127.22.230.1:44133/ using document root <none> and password file <none>
I20260619 03:00:47.496704 7601 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:47.496842 7601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:47.501719 7601 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 03:00:47.504128 7656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.505254 7601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:00:47.505405 7601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "ca2ab7a3cd5e43299ce351e30f8a1cb0"
format_stamp: "Formatted at 2026-06-19 03:00:01 on dist-test-slave-2gms"
server_key: "37289167011ebd386fda7a0013f9885c"
server_key_iv: "7f1a75c4fd23364be69adc676918fd78"
server_key_version: "encryptionkey@0"
I20260619 03:00:47.506232 7601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:47.525072 7601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:47.528853 7601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:47.529101 7601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:47.529982 7601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:47.531209 7601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:47.531323 7601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.531494 7601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:47.531592 7601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.564163 7601 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:40605
I20260619 03:00:47.564211 7768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:40605 every 8 connection(s)
I20260619 03:00:47.565577 7601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:00:47.568856 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7601
I20260619 03:00:47.569447 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:46187
--local_ip_for_outbound_sockets=127.22.230.2
--tserver_master_addrs=127.22.230.62:36975
--webserver_port=37769
--webserver_interface=127.22.230.2
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:00:47.570438 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:47.568462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:35451)
0619 03:00:47.569282 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:47.569578 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
W20260619 03:00:47.571166 7769 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:36975 (0 consecutive failures): Network error: Failed to ping master at 127.22.230.62:36975: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
I20260619 03:00:47.572019 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:47.571683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:36615)
0619 03:00:47.571852 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:47.571923 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 03:00:47.572851 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:47.572531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:37397)
0619 03:00:47.572667 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:47.572744 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 03:00:47.573086 7769 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 03:00:47.877938 7772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:47.878167 7772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:47.878270 7772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:47.885134 7772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:47.885224 7772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:47.885324 7772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:00:47.893996 7772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:46187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=37769
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:47.894999 7772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:47.896088 7772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:47.908524 7777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:47.908640 7780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:47.908640 7778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:47.908931 7772 server_base.cc:1034] running on GCE node
I20260619 03:00:47.909700 7772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:47.910535 7772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:47.911788 7772 hybrid_clock.cc:648] HybridClock initialized: now 1781838047911731 us; error 44 us; skew 500 ppm
I20260619 03:00:47.912060 7772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:47 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838047, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:47.915028 7772 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 03:00:47.916270 7772 webserver.cc:460] Webserver started at http://127.22.230.2:37769/ using document root <none> and password file <none>
I20260619 03:00:47.916890 7772 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:47.917027 7772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:47.921761 7772 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 03:00:47.924199 7787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.925372 7772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:00:47.925521 7772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "e8586a6b38a840f3a65a1cb7880911f8"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "2d2799abb5b1feb0c47f496ffc0ec853"
server_key_iv: "3b8ae8bc3f80a9348df1600baf0439ee"
server_key_version: "encryptionkey@0"
I20260619 03:00:47.926329 7772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:47.959007 7772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:47.963308 7772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:47.963569 7772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:47.964504 7772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:47.965719 7772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:47.965838 7772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.965963 7772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:47.966045 7772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:47.997926 7772 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:46187
I20260619 03:00:47.998003 7899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:46187 every 8 connection(s)
I20260619 03:00:47.999384 7772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:00:48.002242 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7772
I20260619 03:00:48.002904 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:41255
--local_ip_for_outbound_sockets=127.22.230.3
--tserver_master_addrs=127.22.230.62:36975
--webserver_port=35297
--webserver_interface=127.22.230.3
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:00:48.004137 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.002117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:57047)
0619 03:00:48.002949 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.003263 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":337,"threads_started":1}
W20260619 03:00:48.004865 7900 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:36975 (0 consecutive failures): Network error: Failed to ping master at 127.22.230.62:36975: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
I20260619 03:00:48.005641 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.005338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:33395)
0619 03:00:48.005478 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.005548 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 03:00:48.006419 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.006120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:59439)
0619 03:00:48.006248 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.006325 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 03:00:48.006726 7900 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 03:00:48.318987 7903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:00:48.319228 7903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:00:48.319429 7903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:00:48.326202 7903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:00:48.326303 7903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:00:48.326450 7903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:00:48.335431 7903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:41255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=35297
--tserver_master_addrs=127.22.230.62:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:00:48.336524 7903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:00:48.337707 7903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:00:48.350230 7909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:48.350381 7908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:00:48.350186 7911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:00:48.351137 7903 server_base.cc:1034] running on GCE node
I20260619 03:00:48.351727 7903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:00:48.352553 7903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:00:48.353771 7903 hybrid_clock.cc:648] HybridClock initialized: now 1781838048353729 us; error 56 us; skew 500 ppm
I20260619 03:00:48.354063 7903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:00:48 dist-test-slave-2gms krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838048, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:00:48.356942 7903 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 03:00:48.358177 7903 webserver.cc:460] Webserver started at http://127.22.230.3:35297/ using document root <none> and password file <none>
I20260619 03:00:48.358819 7903 fs_manager.cc:359] Metadata directory not provided
I20260619 03:00:48.358915 7903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:00:48.363554 7903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 03:00:48.366070 7918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:48.367355 7903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:00:48.367488 7903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "d25c98e121b1406c9d03c65d13343412"
format_stamp: "Formatted at 2026-06-19 03:00:02 on dist-test-slave-2gms"
server_key: "826100882d2a126856b6c01e88198fdf"
server_key_iv: "98974a79a7feff3dc342db1487ba5472"
server_key_version: "encryptionkey@0"
I20260619 03:00:48.368330 7903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:00:48.381037 7903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:00:48.384441 7903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:00:48.384684 7903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:00:48.385530 7903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:00:48.386780 7903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:00:48.386885 7903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:48.386994 7903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:00:48.387089 7903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:00:48.418788 7903 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:41255
I20260619 03:00:48.418886 8030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:41255 every 8 connection(s)
I20260619 03:00:48.420079 7903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:00:48.424307 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.422632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:56647)
0619 03:00:48.423266 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.423542 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":237,"threads_started":1}
W20260619 03:00:48.424989 8031 heartbeater.cc:644] Failed to heartbeat to 127.22.230.62:36975 (0 consecutive failures): Network error: Failed to ping master at 127.22.230.62:36975: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
I20260619 03:00:48.425690 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.425428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:38195)
0619 03:00:48.425543 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.425600 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 03:00:48.426425 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.426166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:40671)
0619 03:00:48.426267 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.426330 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 03:00:48.426726 8031 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 03:00:48.429152 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 7903
I20260619 03:00:48.574245 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:48.573568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:47159)
0619 03:00:48.573963 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:48.574039 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":208,"threads_started":1}
I20260619 03:00:49.007901 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:49.007156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:37869)
0619 03:00:49.007610 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:49.007689 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
I20260619 03:00:49.427968 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:49.427157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:53465)
0619 03:00:49.427644 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:49.427750 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
I20260619 03:00:49.575635 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:49.575030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:42781)
0619 03:00:49.575426 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:49.575496 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":196,"threads_started":1}
I20260619 03:00:50.009347 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:50.008653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:47127)
0619 03:00:50.009099 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:50.009192 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260619 03:00:50.429471 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:50.428731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:50467)
0619 03:00:50.429253 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:50.429330 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":284,"threads_started":1}
I20260619 03:00:50.576960 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:50.576359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:57587)
0619 03:00:50.576726 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:50.576806 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":193,"threads_started":1}
I20260619 03:00:51.010953 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:51.010257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:45529)
0619 03:00:51.010726 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:51.010800 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260619 03:00:51.430882 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:51.430208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:44993)
0619 03:00:51.430666 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:51.430746 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":203,"threads_started":1}
I20260619 03:00:51.578377 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:51.577699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:50251)
0619 03:00:51.578125 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:51.578196 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":192,"threads_started":1}
I20260619 03:00:52.012382 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:52.011753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:54881)
0619 03:00:52.012154 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:52.012226 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":196,"threads_started":1}
I20260619 03:00:52.432252 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:52.431614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:55063)
0619 03:00:52.432001 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:52.432072 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":193,"threads_started":1}
I20260619 03:00:52.579810 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:52.579194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:32909)
0619 03:00:52.579588 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:52.579658 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":206,"threads_started":1}
I20260619 03:00:53.014060 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:53.013412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:53677)
0619 03:00:53.013823 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:53.013894 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260619 03:00:53.433735 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:53.433113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:42745)
0619 03:00:53.433504 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:53.433574 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":216,"threads_started":1}
I20260619 03:00:53.581238 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:53.580600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:44131)
0619 03:00:53.581016 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:53.581095 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":237,"threads_started":1}
I20260619 03:00:54.015605 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:54.015015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:50249)
0619 03:00:54.015389 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:54.015468 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":200,"threads_started":1}
I20260619 03:00:54.435227 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:54.434478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:39921)
0619 03:00:54.434995 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:54.435074 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":316,"threads_started":1}
I20260619 03:00:54.582842 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:54.581999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:33957)
0619 03:00:54.582427 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:54.582498 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":190,"threads_started":1}
I20260619 03:00:55.017113 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:55.016396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:36351)
0619 03:00:55.016841 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:55.016920 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":204,"threads_started":1}
I20260619 03:00:55.436677 8055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:55.435975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:40569)
0619 03:00:55.436372 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:55.436443 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":174,"threads_started":1}
I20260619 03:00:55.584453 8056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:55.583788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:57075)
0619 03:00:55.584234 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:55.584310 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260619 03:00:56.018764 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:56.018078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:40373)
0619 03:00:56.018469 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:56.018544 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":207,"threads_started":1}
I20260619 03:00:56.438072 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:56.437394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:35331)
0619 03:00:56.437862 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:56.437940 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260619 03:00:56.585870 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:56.585225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:56485)
0619 03:00:56.585611 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:56.585680 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":191,"threads_started":1}
I20260619 03:00:57.020591 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:57.019481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:51321)
0619 03:00:57.019964 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:57.020072 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260619 03:00:57.439594 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:57.438966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:39463)
0619 03:00:57.439372 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:57.439442 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260619 03:00:57.587452 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:57.586832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:34137)
0619 03:00:57.587234 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:57.587314 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":223,"threads_started":1}
I20260619 03:00:58.022432 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:58.021643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:60497)
0619 03:00:58.022189 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:58.022283 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":365,"threads_started":1}
I20260619 03:00:58.441131 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:58.440505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:33711)
0619 03:00:58.440903 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:58.440977 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":213,"threads_started":1}
I20260619 03:00:58.588829 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:58.588190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:44231)
0619 03:00:58.588590 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:58.588670 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":224,"threads_started":1}
I20260619 03:00:59.023887 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:59.023177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:39623)
0619 03:00:59.023632 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:59.023714 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":240,"threads_started":1}
I20260619 03:00:59.442497 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:59.441892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:38089)
0619 03:00:59.442269 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:59.442339 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":189,"threads_started":1}
I20260619 03:00:59.590163 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:00:59.589571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:56771)
0619 03:00:59.589956 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 03:00:59.590028 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":198,"threads_started":1}
I20260619 03:01:00.025394 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:00.024760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:60353)
0619 03:01:00.025169 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:00.025245 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":220,"threads_started":1}
I20260619 03:01:00.443929 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:00.443326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:48841)
0619 03:01:00.443714 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:00.443792 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":202,"threads_started":1}
I20260619 03:01:00.592029 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:00.591178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:47185)
0619 03:01:00.591673 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:00.591788 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260619 03:01:01.027347 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:01.026502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:45107)
0619 03:01:01.027070 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:01.027170 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":337,"threads_started":1}
I20260619 03:01:01.445614 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:01.444910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:44233)
0619 03:01:01.445339 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:01.445451 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260619 03:01:01.593641 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:01.592969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:52933)
0619 03:01:01.593383 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:01.593476 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":204,"threads_started":1}
I20260619 03:01:02.028980 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:02.028327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:51195)
0619 03:01:02.028731 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:02.028808 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":215,"threads_started":1}
I20260619 03:01:02.447151 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:02.446471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:60605)
0619 03:01:02.446894 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:02.446986 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260619 03:01:02.595304 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:02.594666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.1:43693)
0619 03:01:02.595047 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:02.595145 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":196,"threads_started":1}
I20260619 03:01:03.030858 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:03.030139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.2:39499)
0619 03:01:03.030518 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:03.030654 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":197,"threads_started":1}
I20260619 03:01:03.441047 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7601
I20260619 03:01:03.448925 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:03.448130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36975 (local address 127.22.230.3:55349)
0619 03:01:03.448595 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:03.448701 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.62:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260619 03:01:03.462961 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:03.442048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.22.230.1:40605)
0619 03:01:03.442738 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:03.442758 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:03.442809 (+ 51us) server_negotiation.cc:407] Connection header received
0619 03:01:03.444375 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:03.444384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:03.444794 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:03.445153 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:03.446716 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:03.448060 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:03.449693 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:03.450394 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:03.450525 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:03.457363 (+ 6838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:03.457372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:03.457387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:03.457427 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:03.460166 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:03.460778 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:03.460784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:03.460790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:03.460960 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:03.461259 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:03.461262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:03.461264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:03.461742 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:03.461983 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:03.462288 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 03:01:03.462684 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":284,"threads_started":1}
I20260619 03:01:03.464406 7764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:03.546473 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7601
I20260619 03:01:03.563746 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7772
I20260619 03:01:03.580636 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:03.564585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.22.230.2:46187)
0619 03:01:03.565091 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:03.565105 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:03.565142 (+ 37us) server_negotiation.cc:407] Connection header received
0619 03:01:03.566739 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:03.566748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:03.567163 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:03.567512 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:03.568943 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:03.570224 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:03.571615 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:03.572263 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:03.572466 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:03.574949 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:03.574958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:03.574972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:03.575008 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:03.577589 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:03.578245 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:03.578253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:03.578266 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:03.578429 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:03.578820 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:03.578825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:03.578828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:03.579291 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:03.579598 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:03.579977 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 03:01:03.580353 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260619 03:01:03.581931 7895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:03.666527 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7772
I20260619 03:01:03.685675 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 7903
I20260619 03:01:03.702739 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:03.686656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.22.230.3:41255)
0619 03:01:03.687245 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:03.687259 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:03.687299 (+ 40us) server_negotiation.cc:407] Connection header received
0619 03:01:03.688948 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:03.688957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:03.689360 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:03.689706 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:03.691215 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:03.692430 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:03.693688 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:03.694314 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:03.694522 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:03.696989 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:03.696996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:03.697010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:03.697047 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:03.699642 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:03.700314 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:03.700320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:03.700325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:03.700488 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:03.700866 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:03.700870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:03.700872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:03.701335 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:03.701671 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:03.702015 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 03:01:03.702385 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":255,"threads_started":1}
I20260619 03:01:03.703998 8026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:03.786343 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 7903
2026-06-19T03:01:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64068 ms)
[----------] 30 tests from SecurityITest (288911 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:03 dist-test-slave-2gms krb5kdc[8088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:03 dist-test-slave-2gms krb5kdc[8088](info): set up 2 sockets
Jun 19 03:01:03 dist-test-slave-2gms krb5kdc[8088](info): commencing operation
krb5kdc: starting...
W20260619 03:01:04.866480 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:05 dist-test-slave-2gms krb5kdc[8088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:05.083187 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36599
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:05.389076 8101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:05.389370 8101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:05.389559 8101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:05.396382 8101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:05.396517 8101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:05.396620 8101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:05.396723 8101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:05.396807 8101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:05.405933 8101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:36599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:05.407130 8101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:05.408356 8101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:05.419297 8106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:05.419369 8109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:05.419369 8107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:05.420262 8101 server_base.cc:1034] running on GCE node
I20260619 03:01:05.420976 8101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:05.422307 8101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:05.423535 8101 hybrid_clock.cc:648] HybridClock initialized: now 1781838065423497 us; error 41 us; skew 500 ppm
I20260619 03:01:05.423862 8101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:05 dist-test-slave-2gms krb5kdc[8088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:05.427341 8101 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:05.428979 8101 webserver.cc:460] Webserver started at http://127.22.230.62:46757/ using document root <none> and password file <none>
I20260619 03:01:05.429615 8101 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:05.429736 8101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:05.430016 8101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:05.432746 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "2355bd69c36e43e08271150382424af6"
format_stamp: "Formatted at 2026-06-19 03:01:05 on dist-test-slave-2gms"
server_key: "d8275ea00c42fbe8692ead34f0e6f8dd"
server_key_iv: "9b75188f64ddd143d232b4fe47cd573a"
server_key_version: "encryptionkey@0"
I20260619 03:01:05.433454 8101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "2355bd69c36e43e08271150382424af6"
format_stamp: "Formatted at 2026-06-19 03:01:05 on dist-test-slave-2gms"
server_key: "d8275ea00c42fbe8692ead34f0e6f8dd"
server_key_iv: "9b75188f64ddd143d232b4fe47cd573a"
server_key_version: "encryptionkey@0"
I20260619 03:01:05.438169 8101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 03:01:05.441202 8116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:05.442351 8101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:05.442526 8101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "2355bd69c36e43e08271150382424af6"
format_stamp: "Formatted at 2026-06-19 03:01:05 on dist-test-slave-2gms"
server_key: "d8275ea00c42fbe8692ead34f0e6f8dd"
server_key_iv: "9b75188f64ddd143d232b4fe47cd573a"
server_key_version: "encryptionkey@0"
I20260619 03:01:05.442752 8101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:05.459630 8101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:05.463038 8101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:05.463272 8101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:05.480633 8101 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:36599
I20260619 03:01:05.480660 8167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:36599 every 8 connection(s)
I20260619 03:01:05.481822 8101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:05.482955 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8101
I20260619 03:01:05.483171 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:05.483810 23448 external_mini_cluster.cc:1428] Setting key f20d748a2668d1c24304871edaccd2f7
I20260619 03:01:05.485846 8168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 03:01:05 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:05.498260 8168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6: Bootstrap starting.
I20260619 03:01:05.501431 8168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:05.502384 8168 log.cc:824] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:05.505152 8168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6: No bootstrap required, opened a new log
I20260619 03:01:05.505281 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:05.485566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.22.230.62:36599)
0619 03:01:05.486306 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:05.486316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:05.486353 (+ 37us) server_negotiation.cc:407] Connection header received
0619 03:01:05.487974 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:05.487992 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:05.488330 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:05.488606 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:05.490354 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:05.491629 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:05.493326 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:05.493893 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:05.494013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:05.498673 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:05.498681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:05.498693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:05.498728 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:05.501430 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:05.502103 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:05.502117 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:05.502122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:05.502285 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:05.502713 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:05.502718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:05.502720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:05.503389 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:05.503660 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:05.504054 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 03:01:05.504322 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":377,"threads_started":1}
I20260619 03:01:05.512003 8168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2355bd69c36e43e08271150382424af6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36599 } }
I20260619 03:01:05.512327 8168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:05.512454 8168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2355bd69c36e43e08271150382424af6, State: Initialized, Role: FOLLOWER
I20260619 03:01:05.512984 8168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2355bd69c36e43e08271150382424af6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36599 } }
I20260619 03:01:05.513197 8168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:05.513330 8168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:05.513489 8168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:05.515446 8168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2355bd69c36e43e08271150382424af6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36599 } }
I20260619 03:01:05.515906 8168 leader_election.cc:302] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2355bd69c36e43e08271150382424af6; no voters:
I20260619 03:01:05.516510 8168 leader_election.cc:288] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:05.516600 8173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:05.517226 8173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [term 1 LEADER]: Becoming Leader. State: Replica: 2355bd69c36e43e08271150382424af6, State: Running, Role: LEADER
I20260619 03:01:05.517741 8173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2355bd69c36e43e08271150382424af6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36599 } }
I20260619 03:01:05.518436 8168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:05.520633 8174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2355bd69c36e43e08271150382424af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2355bd69c36e43e08271150382424af6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36599 } } }
I20260619 03:01:05.521044 8175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2355bd69c36e43e08271150382424af6. Latest consensus state: current_term: 1 leader_uuid: "2355bd69c36e43e08271150382424af6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2355bd69c36e43e08271150382424af6" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36599 } } }
I20260619 03:01:05.521377 8174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:05.521414 8175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:05.527388 8180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:05.532317 8180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:05.540163 8180 catalog_manager.cc:1347] Generated new cluster ID: 0fcfd4c2408343bca554d02c1b85f58d
I20260619 03:01:05.540269 8180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:05.559140 8180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:05.560328 8180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:05.571681 8180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6: Generated new TSK 0
I20260619 03:01:05.572444 8180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 03:01:05.636035 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36599
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:05.962919 8194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:05.963150 8194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:05.963233 8194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:05.969959 8194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:05.970049 8194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:05.970144 8194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:01:05.978533 8194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:05.979535 8194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:05.980618 8194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:05.992688 8200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:05.992761 8199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:05.992645 8202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:05.993271 8194 server_base.cc:1034] running on GCE node
I20260619 03:01:05.993769 8194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:05.994612 8194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:05.995821 8194 hybrid_clock.cc:648] HybridClock initialized: now 1781838065995767 us; error 51 us; skew 500 ppm
I20260619 03:01:05.996134 8194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:05 dist-test-slave-2gms krb5kdc[8088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:05.998914 8194 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 03:01:06.000188 8194 webserver.cc:460] Webserver started at http://127.22.230.1:45655/ using document root <none> and password file <none>
I20260619 03:01:06.000789 8194 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:06.000917 8194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:06.001196 8194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:06.003934 8194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "35fbc8a315c34714bd06622000bb8ab1"
format_stamp: "Formatted at 2026-06-19 03:01:05 on dist-test-slave-2gms"
server_key: "e6d8981dcc11613cb6465092ac3195e2"
server_key_iv: "946730f41f400e57892c296f929c5140"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.004619 8194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "35fbc8a315c34714bd06622000bb8ab1"
format_stamp: "Formatted at 2026-06-19 03:01:05 on dist-test-slave-2gms"
server_key: "e6d8981dcc11613cb6465092ac3195e2"
server_key_iv: "946730f41f400e57892c296f929c5140"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.009470 8194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 03:01:06.012415 8209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.013590 8194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:06.013751 8194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "35fbc8a315c34714bd06622000bb8ab1"
format_stamp: "Formatted at 2026-06-19 03:01:05 on dist-test-slave-2gms"
server_key: "e6d8981dcc11613cb6465092ac3195e2"
server_key_iv: "946730f41f400e57892c296f929c5140"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.013935 8194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:06.033944 8194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:06.037478 8194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:06.037737 8194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:06.038637 8194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:06.039763 8194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:06.039863 8194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.039999 8194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:06.040066 8194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.076879 8194 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:33349
I20260619 03:01:06.076920 8321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:33349 every 8 connection(s)
I20260619 03:01:06.078030 8194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:01:06.080029 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8194
I20260619 03:01:06.080225 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:01:06.081036 23448 external_mini_cluster.cc:1428] Setting key ccf2b237e63b4b169c6c7ab8861bbfc8
Jun 19 03:01:06 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:06.098323 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:06.081700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:55317 (local address 127.22.230.62:36599)
0619 03:01:06.082898 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:06.082903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:06.082921 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:01:06.084336 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:06.084345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:06.084431 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:06.084544 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:06.085927 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.087098 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.088511 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.089179 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.089317 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:06.092556 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:06.092562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:06.092569 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:06.092610 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:06.095265 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:06.096191 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:06.096195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:06.096197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:06.096331 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:06.096812 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:06.096815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:06.096817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:06.097051 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:06.097157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:06.097956 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 03:01:06.098093 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":1001,"threads_started":1}
I20260619 03:01:06.099788 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:06.081134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36599 (local address 127.22.230.1:55317)
0619 03:01:06.082021 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:06.082084 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:06.084050 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:06.084725 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:06.084752 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:06.085218 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:06.085744 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:06.085765 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.087237 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.087242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:06.088364 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:06.088387 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.089390 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.089395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:06.089573 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:06.090200 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:06.090220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:06.092364 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:06.095398 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:06.095409 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:06.095426 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:06.096075 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:06.096439 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:06.096447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:06.096453 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:06.096706 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:06.097173 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:06.097184 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:06.097596 (+ 412us) client_negotiation.cc:769] Sending connection context
0619 03:01:06.097911 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 03:01:06.098408 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
I20260619 03:01:06.101240 8322 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36599
I20260619 03:01:06.101517 8322 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:06.102201 8322 heartbeater.cc:511] Master 127.22.230.62:36599 requested a full tablet report, sending...
I20260619 03:01:06.104151 8133 ts_manager.cc:194] Registered new tserver with Master: 35fbc8a315c34714bd06622000bb8ab1 (127.22.230.1:33349)
I20260619 03:01:06.105564 8133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:55317
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 03:01:06.135088 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36599
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:06.436025 8328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:06.436296 8328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:06.436445 8328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:06.443288 8328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:06.443425 8328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:06.443585 8328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:01:06.453537 8328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:06.454842 8328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:06.456034 8328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:06.468724 8336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:06.468695 8334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:06.468755 8333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:06.469450 8328 server_base.cc:1034] running on GCE node
I20260619 03:01:06.469998 8328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:06.470914 8328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:06.472127 8328 hybrid_clock.cc:648] HybridClock initialized: now 1781838066472091 us; error 49 us; skew 500 ppm
I20260619 03:01:06.472414 8328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:06 dist-test-slave-2gms krb5kdc[8088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:06.475499 8328 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 03:01:06.476887 8328 webserver.cc:460] Webserver started at http://127.22.230.2:44459/ using document root <none> and password file <none>
I20260619 03:01:06.477485 8328 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:06.477619 8328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:06.477916 8328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:06.480567 8328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "0733654bd3bc4ecc869c9e03f77b92b6"
format_stamp: "Formatted at 2026-06-19 03:01:06 on dist-test-slave-2gms"
server_key: "f2f0c9e8fecd2c0bcd9233ea0b42e7af"
server_key_iv: "09baea12a7e0dc532ff15036640b9533"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.481284 8328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "0733654bd3bc4ecc869c9e03f77b92b6"
format_stamp: "Formatted at 2026-06-19 03:01:06 on dist-test-slave-2gms"
server_key: "f2f0c9e8fecd2c0bcd9233ea0b42e7af"
server_key_iv: "09baea12a7e0dc532ff15036640b9533"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.485954 8328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 03:01:06.489184 8343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.490338 8328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:06.490504 8328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "0733654bd3bc4ecc869c9e03f77b92b6"
format_stamp: "Formatted at 2026-06-19 03:01:06 on dist-test-slave-2gms"
server_key: "f2f0c9e8fecd2c0bcd9233ea0b42e7af"
server_key_iv: "09baea12a7e0dc532ff15036640b9533"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.490734 8328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:06.527792 8328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:06.531821 8328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:06.532127 8328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:06.533036 8328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:06.534204 8328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:06.534297 8328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.534435 8328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:06.534502 8328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.564164 8328 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:36305
I20260619 03:01:06.564239 8455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:36305 every 8 connection(s)
I20260619 03:01:06.565346 8328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:01:06.566450 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8328
I20260619 03:01:06.566671 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:01:06.567139 23448 external_mini_cluster.cc:1428] Setting key d8dae3c2d4e70621e7b819c02168cd85
Jun 19 03:01:06 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:06.584432 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:06.567977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:50731 (local address 127.22.230.62:36599)
0619 03:01:06.568138 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:06.568143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:06.569320 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 03:01:06.571149 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:06.571154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:06.571241 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:06.571354 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:06.572774 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.573436 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.574979 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.575365 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.575498 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:06.578890 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:06.578894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:06.578898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:06.578958 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:06.581380 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:06.582244 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:06.582247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:06.582249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:06.582363 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:06.582859 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:06.582862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:06.582864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:06.583109 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:06.583233 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:06.584059 (+ 826us) server_negotiation.cc:299] Negotiation successful
0619 03:01:06.584212 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 03:01:06.585906 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:06.568410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36599 (local address 127.22.230.2:50731)
0619 03:01:06.569163 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:06.569208 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:06.570889 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:06.571560 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:06.571568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:06.572148 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:06.572626 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:06.572642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.573584 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.573588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:06.574810 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:06.574821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:06.575677 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:06.575684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:06.575868 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:06.576550 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:06.576571 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:06.578696 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:06.581511 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:06.581521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:06.581535 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:06.582131 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:06.582461 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:06.582468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:06.582474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:06.582751 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:06.583245 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:06.583254 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:06.583649 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 03:01:06.583970 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 03:01:06.584527 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":283,"threads_started":1}
I20260619 03:01:06.587275 8456 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36599
I20260619 03:01:06.587545 8456 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:06.588202 8456 heartbeater.cc:511] Master 127.22.230.62:36599 requested a full tablet report, sending...
I20260619 03:01:06.589532 8133 ts_manager.cc:194] Registered new tserver with Master: 0733654bd3bc4ecc869c9e03f77b92b6 (127.22.230.2:36305)
I20260619 03:01:06.590304 8133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:50731
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 03:01:06.619283 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36599
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:06.921175 8461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:06.921409 8461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:06.921492 8461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:06.928154 8461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:06.928246 8461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:06.928354 8461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:01:06.936683 8461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:06.937799 8461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:06.938946 8461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:06.951220 8467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:06.951319 8466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:06.951195 8469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:06.951975 8461 server_base.cc:1034] running on GCE node
I20260619 03:01:06.952431 8461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:06.953162 8461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:06.954346 8461 hybrid_clock.cc:648] HybridClock initialized: now 1781838066954316 us; error 41 us; skew 500 ppm
I20260619 03:01:06.954699 8461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:06 dist-test-slave-2gms krb5kdc[8088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:06.961269 8461 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 03:01:06.962694 8461 webserver.cc:460] Webserver started at http://127.22.230.3:36991/ using document root <none> and password file <none>
I20260619 03:01:06.963294 8461 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:06.963428 8461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:06.963711 8461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:06.966301 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "35e0882c86114cdb8f053b18053695c0"
format_stamp: "Formatted at 2026-06-19 03:01:06 on dist-test-slave-2gms"
server_key: "ba9b2041192d000e9fafd3ae714abd62"
server_key_iv: "397e6150ca726dcfd1e3cf4006ca3b70"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.967043 8461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "35e0882c86114cdb8f053b18053695c0"
format_stamp: "Formatted at 2026-06-19 03:01:06 on dist-test-slave-2gms"
server_key: "ba9b2041192d000e9fafd3ae714abd62"
server_key_iv: "397e6150ca726dcfd1e3cf4006ca3b70"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.971895 8461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 03:01:06.974956 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.976138 8461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 03:01:06.976313 8461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "35e0882c86114cdb8f053b18053695c0"
format_stamp: "Formatted at 2026-06-19 03:01:06 on dist-test-slave-2gms"
server_key: "ba9b2041192d000e9fafd3ae714abd62"
server_key_iv: "397e6150ca726dcfd1e3cf4006ca3b70"
server_key_version: "encryptionkey@0"
I20260619 03:01:06.976505 8461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:06.990175 8461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:06.993656 8461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:06.993942 8461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:06.994899 8461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:06.996075 8461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:06.996176 8461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:06.996300 8461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:06.996379 8461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:07.028277 8461 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:40717
I20260619 03:01:07.028311 8588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:40717 every 8 connection(s)
I20260619 03:01:07.029488 8461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:01:07.038774 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8461
I20260619 03:01:07.038964 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:01:07.039501 23448 external_mini_cluster.cc:1428] Setting key 90b10a6b33072a24b585f9845b609748
Jun 19 03:01:07 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838066, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:07.047868 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.031841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:39557 (local address 127.22.230.62:36599)
0619 03:01:07.032032 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.032037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.033105 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 03:01:07.034773 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.034778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.034856 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.034938 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:07.036347 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.037053 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.038429 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.038859 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.038987 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.042598 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:07.042606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:07.042614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:07.042671 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:07.045285 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.045933 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.045936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.045939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.046074 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.046443 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.046446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.046447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.046775 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:07.046880 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.047475 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.047619 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 03:01:07.049147 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.032316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36599 (local address 127.22.230.3:39557)
0619 03:01:07.032958 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.032992 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.034488 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.035124 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.035131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.035585 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:07.036137 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.036152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.037173 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.037177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.038313 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.038324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.039082 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.039090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.039363 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.039993 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:07.040022 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:07.042397 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:07.045407 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:07.045413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:07.045423 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:07.045841 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:07.046173 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:07.046176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:07.046178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:07.046351 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:07.046893 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:07.046904 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:07.047181 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.047383 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.047700 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":246,"threads_started":1}
I20260619 03:01:07.050828 8589 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36599
I20260619 03:01:07.051095 8589 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:07.052146 8589 heartbeater.cc:511] Master 127.22.230.62:36599 requested a full tablet report, sending...
I20260619 03:01:07.054073 8133 ts_manager.cc:194] Registered new tserver with Master: 35e0882c86114cdb8f053b18053695c0 (127.22.230.3:40717)
I20260619 03:01:07.055552 8133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:39557
I20260619 03:01:07.056152 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:01:07.083845 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.064553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.22.230.62:36599)
0619 03:01:07.064727 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.064733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.065040 (+ 307us) server_negotiation.cc:407] Connection header received
0619 03:01:07.065203 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.065206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.065262 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.065339 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:07.067291 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.068289 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.070369 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.071142 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.071310 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.074387 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:07.074393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:07.074397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:07.074436 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:07.080587 (+ 6151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.081414 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.081420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.081425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.081582 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.081987 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.081994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.081998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.082494 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:07.082767 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.083095 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.083306 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 03:01:07.088493 8133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:01:07.095714 8133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:01:07.108278 8322 heartbeater.cc:503] Master 127.22.230.62:36599 was elected leader, sending a full tablet report...
I20260619 03:01:07.118384 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.107460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40717 (local address 127.0.0.1:36536)
0619 03:01:07.108720 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.108768 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.108973 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.109695 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.109703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.109732 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.110603 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.110618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.112667 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.112671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.115479 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.115493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.117265 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.117271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.117470 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.117521 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.117655 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.117835 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":110,"thread_start_us":1174,"threads_started":1}
I20260619 03:01:07.119237 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.107647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.22.230.3:40717)
0619 03:01:07.108440 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.108450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.108875 (+ 425us) server_negotiation.cc:407] Connection header received
0619 03:01:07.109183 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.109195 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.109467 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.109906 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.110787 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.112516 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.115642 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.117060 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.117170 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.117428 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.118652 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.118860 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":459,"threads_started":1}
I20260619 03:01:07.119740 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.109790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:36305 (local address 127.0.0.1:56682)
0619 03:01:07.111744 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.111762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.111871 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.112669 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.112674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.112696 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.113201 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.113214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.115027 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.115034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.116957 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.116972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.118944 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.118950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.119120 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.119150 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.119237 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.119388 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"mutex_wait_us":3,"thread_start_us":1661,"threads_started":1}
I20260619 03:01:07.120121 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.107809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:33349 (local address 127.0.0.1:38346)
0619 03:01:07.109509 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.109526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.109634 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.110799 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.110806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.110829 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.111189 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.111197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.113100 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.113107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.115479 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.115493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.119301 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.119308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.119488 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.119530 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.119626 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.119775 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"mutex_wait_us":60,"thread_start_us":1444,"threads_started":1}
I20260619 03:01:07.120755 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.108297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.22.230.1:33349)
0619 03:01:07.110093 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.110101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.110127 (+ 26us) server_negotiation.cc:407] Connection header received
0619 03:01:07.110211 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.110218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.110435 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.110660 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.111354 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.112957 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.117514 (+ 4557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.119094 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.119254 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.119475 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.120349 (+ 874us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.120514 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":1639,"threads_started":1}
I20260619 03:01:07.120846 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.110035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.22.230.2:36305)
0619 03:01:07.110630 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.110641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.111987 (+ 1346us) server_negotiation.cc:407] Connection header received
0619 03:01:07.112079 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.112086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.112282 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.112472 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.113374 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.114890 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.117111 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.118616 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.118746 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.120126 (+ 1380us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.120324 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.120529 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":268,"threads_started":1}
I20260619 03:01:07.123131 8524 tablet_service.cc:1467] Processing CreateTablet for tablet f496f280117a4beebadf213ff2b40208 (DEFAULT_TABLE table=test-table [id=86a2ef49a8594d7c9b0b66d9cb6e414a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:07.124672 8524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f496f280117a4beebadf213ff2b40208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:07.125686 8391 tablet_service.cc:1467] Processing CreateTablet for tablet f496f280117a4beebadf213ff2b40208 (DEFAULT_TABLE table=test-table [id=86a2ef49a8594d7c9b0b66d9cb6e414a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:07.125726 8257 tablet_service.cc:1467] Processing CreateTablet for tablet f496f280117a4beebadf213ff2b40208 (DEFAULT_TABLE table=test-table [id=86a2ef49a8594d7c9b0b66d9cb6e414a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:07.126859 8391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f496f280117a4beebadf213ff2b40208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:07.127240 8257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f496f280117a4beebadf213ff2b40208. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:07.137955 8607 tablet_bootstrap.cc:492] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Bootstrap starting.
I20260619 03:01:07.139899 8608 tablet_bootstrap.cc:492] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Bootstrap starting.
I20260619 03:01:07.141588 8607 tablet_bootstrap.cc:654] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:07.142858 8607 log.cc:824] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:07.143215 8608 tablet_bootstrap.cc:654] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:07.143275 8609 tablet_bootstrap.cc:492] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Bootstrap starting.
I20260619 03:01:07.144585 8608 log.cc:824] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:07.146739 8609 tablet_bootstrap.cc:654] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:07.147647 8609 log.cc:824] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:07.147928 8607 tablet_bootstrap.cc:492] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: No bootstrap required, opened a new log
I20260619 03:01:07.148283 8607 ts_tablet_manager.cc:1397] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 03:01:07.149964 8609 tablet_bootstrap.cc:492] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: No bootstrap required, opened a new log
I20260619 03:01:07.150205 8609 ts_tablet_manager.cc:1397] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 03:01:07.156945 8608 tablet_bootstrap.cc:492] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: No bootstrap required, opened a new log
I20260619 03:01:07.157182 8608 ts_tablet_manager.cc:1397] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260619 03:01:07.157591 8609 raft_consensus.cc:348] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.157928 8609 raft_consensus.cc:374] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:07.158048 8609 raft_consensus.cc:729] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fbc8a315c34714bd06622000bb8ab1, State: Initialized, Role: FOLLOWER
I20260619 03:01:07.158687 8609 consensus_queue.cc:260] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.160022 8607 raft_consensus.cc:348] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.160408 8607 raft_consensus.cc:374] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:07.160496 8607 raft_consensus.cc:729] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0733654bd3bc4ecc869c9e03f77b92b6, State: Initialized, Role: FOLLOWER
I20260619 03:01:07.161161 8607 consensus_queue.cc:260] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.161525 8609 ts_tablet_manager.cc:1428] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260619 03:01:07.167157 8456 heartbeater.cc:503] Master 127.22.230.62:36599 was elected leader, sending a full tablet report...
I20260619 03:01:07.167814 8607 ts_tablet_manager.cc:1428] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Time spent starting tablet: real 0.019s user 0.013s sys 0.007s
I20260619 03:01:07.168758 8608 raft_consensus.cc:348] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.169081 8608 raft_consensus.cc:374] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:07.169210 8608 raft_consensus.cc:729] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e0882c86114cdb8f053b18053695c0, State: Initialized, Role: FOLLOWER
I20260619 03:01:07.169705 8608 consensus_queue.cc:260] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.170747 8589 heartbeater.cc:503] Master 127.22.230.62:36599 was elected leader, sending a full tablet report...
I20260619 03:01:07.171321 8608 ts_tablet_manager.cc:1428] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260619 03:01:07.281109 8590 tablet.cc:2369] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:01:07.317198 8457 tablet.cc:2369] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:01:07.330335 8323 tablet.cc:2369] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:01:07.366667 8615 raft_consensus.cc:482] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:07.367023 8615 raft_consensus.cc:504] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.368630 8615 leader_election.cc:288] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35fbc8a315c34714bd06622000bb8ab1 (127.22.230.1:33349), 0733654bd3bc4ecc869c9e03f77b92b6 (127.22.230.2:36305)
I20260619 03:01:07.372713 8613 raft_consensus.cc:482] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:07.373046 8613 raft_consensus.cc:504] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.374516 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.369083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:33349 (local address 127.22.230.3:36223)
0619 03:01:07.369293 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.369313 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.369436 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.369836 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.369841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.369869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.370074 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.370083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.371194 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.371199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.372464 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.372478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.373809 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.373816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.374005 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.374028 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.374125 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.374248 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260619 03:01:07.374526 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.369212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:36223 (local address 127.22.230.1:33349)
0619 03:01:07.369395 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.369399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.369417 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:01:07.369582 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.369587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.369675 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.369803 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.370218 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.371038 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.372644 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.373663 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.373832 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.373908 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.374163 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.374276 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 03:01:07.376593 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.369528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:36305 (local address 127.22.230.3:36667)
0619 03:01:07.370314 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.370327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.370463 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.370819 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.370823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.370846 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.371187 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.371198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.372159 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.372164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.374067 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.374081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.375650 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.375657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.375829 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.375865 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.375957 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.376137 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":452,"threads_started":1}
I20260619 03:01:07.376839 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.369774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:36667 (local address 127.22.230.2:36305)
0619 03:01:07.369975 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.369981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.370423 (+ 442us) server_negotiation.cc:407] Connection header received
0619 03:01:07.370626 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.370635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.370706 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.370803 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.371348 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.372011 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.374232 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.375462 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.375617 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.375688 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.376305 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.376479 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 03:01:07.377887 8411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496f280117a4beebadf213ff2b40208" candidate_uuid: "35e0882c86114cdb8f053b18053695c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" is_pre_election: true
I20260619 03:01:07.378512 8411 raft_consensus.cc:2454] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e0882c86114cdb8f053b18053695c0 in term 0.
I20260619 03:01:07.379354 8478 leader_election.cc:302] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0733654bd3bc4ecc869c9e03f77b92b6, 35e0882c86114cdb8f053b18053695c0; no voters:
I20260619 03:01:07.380100 8277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496f280117a4beebadf213ff2b40208" candidate_uuid: "35e0882c86114cdb8f053b18053695c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fbc8a315c34714bd06622000bb8ab1" is_pre_election: true
I20260619 03:01:07.380645 8277 raft_consensus.cc:2454] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e0882c86114cdb8f053b18053695c0 in term 0.
I20260619 03:01:07.381587 8613 leader_election.cc:288] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35e0882c86114cdb8f053b18053695c0 (127.22.230.3:40717), 0733654bd3bc4ecc869c9e03f77b92b6 (127.22.230.2:36305)
I20260619 03:01:07.381850 8615 raft_consensus.cc:2781] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:01:07.382023 8615 raft_consensus.cc:482] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:01:07.382143 8615 raft_consensus.cc:3037] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:07.386255 8615 raft_consensus.cc:504] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.387476 8277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496f280117a4beebadf213ff2b40208" candidate_uuid: "35e0882c86114cdb8f053b18053695c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fbc8a315c34714bd06622000bb8ab1"
I20260619 03:01:07.387674 8277 raft_consensus.cc:3037] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:07.387996 8411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496f280117a4beebadf213ff2b40208" candidate_uuid: "35e0882c86114cdb8f053b18053695c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0733654bd3bc4ecc869c9e03f77b92b6"
I20260619 03:01:07.388222 8411 raft_consensus.cc:3037] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:07.388820 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.375330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:55121 (local address 127.22.230.3:40717)
0619 03:01:07.375519 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.375543 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.380009 (+ 4466us) server_negotiation.cc:407] Connection header received
0619 03:01:07.380307 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.380314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.380394 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.380457 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.381412 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.382843 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.386143 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.387737 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.387856 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.387939 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.388482 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.388598 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 03:01:07.389436 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.375731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:40717 (local address 127.22.230.1:55121)
0619 03:01:07.379887 (+ 4156us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.379919 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.380127 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.380729 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.380737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.380773 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.381236 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.381251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.383060 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.383069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.385921 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.385940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.388131 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.388139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.388308 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.388340 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.388971 (+ 631us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.389095 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3977,"threads_started":1}
I20260619 03:01:07.390960 8277 raft_consensus.cc:2454] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e0882c86114cdb8f053b18053695c0 in term 1.
I20260619 03:01:07.391336 8478 leader_election.cc:302] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e0882c86114cdb8f053b18053695c0, 35fbc8a315c34714bd06622000bb8ab1; no voters:
I20260619 03:01:07.392295 8411 raft_consensus.cc:2454] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e0882c86114cdb8f053b18053695c0 in term 1.
I20260619 03:01:07.393136 8620 raft_consensus.cc:2781] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:07.393338 8615 leader_election.cc:288] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [CANDIDATE]: Term 1 election: Requested vote from peers 35fbc8a315c34714bd06622000bb8ab1 (127.22.230.1:33349), 0733654bd3bc4ecc869c9e03f77b92b6 (127.22.230.2:36305)
I20260619 03:01:07.393652 8544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496f280117a4beebadf213ff2b40208" candidate_uuid: "35fbc8a315c34714bd06622000bb8ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e0882c86114cdb8f053b18053695c0" is_pre_election: true
I20260619 03:01:07.394477 8620 raft_consensus.cc:686] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 1 LEADER]: Becoming Leader. State: Replica: 35e0882c86114cdb8f053b18053695c0, State: Running, Role: LEADER
I20260619 03:01:07.395052 8620 consensus_queue.cc:237] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } }
I20260619 03:01:07.396015 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.377179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:36305 (local address 127.22.230.1:34457)
0619 03:01:07.389623 (+ 12444us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:07.389639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:07.389789 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:07.390234 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:07.390239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:07.390261 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:07.390414 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.390423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.392041 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.392049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.393940 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:07.393954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.395326 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.395332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:07.395515 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.395537 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 03:01:07.395640 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 03:01:07.395754 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12338,"thread_start_us":2600,"threads_started":1}
I20260619 03:01:07.396040 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.375337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:34457 (local address 127.22.230.2:36305)
0619 03:01:07.379284 (+ 3947us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.379292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.389699 (+ 10407us) server_negotiation.cc:407] Connection header received
0619 03:01:07.389941 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.389949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.390020 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.390077 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:07.390544 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.391833 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.394099 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.395135 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.395323 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.395388 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.395664 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.395778 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3828,"thread_start_us":17032,"threads_started":1}
I20260619 03:01:07.396706 8411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f496f280117a4beebadf213ff2b40208" candidate_uuid: "35fbc8a315c34714bd06622000bb8ab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" is_pre_election: true
I20260619 03:01:07.396996 8411 raft_consensus.cc:2379] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35fbc8a315c34714bd06622000bb8ab1 in current term 1: Already voted for candidate 35e0882c86114cdb8f053b18053695c0 in this term.
I20260619 03:01:07.399089 8212 leader_election.cc:302] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35fbc8a315c34714bd06622000bb8ab1; no voters: 0733654bd3bc4ecc869c9e03f77b92b6, 35e0882c86114cdb8f053b18053695c0
I20260619 03:01:07.399453 8613 raft_consensus.cc:2726] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 03:01:07.399909 8132 catalog_manager.cc:5495] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e0882c86114cdb8f053b18053695c0 (127.22.230.3). New cstate: current_term: 1 leader_uuid: "35e0882c86114cdb8f053b18053695c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35e0882c86114cdb8f053b18053695c0" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 40717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 } health_report { overall_health: UNKNOWN } } }
I20260619 03:01:07.414639 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.409833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.22.230.3:40717)
0619 03:01:07.409969 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.409973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.410106 (+ 133us) server_negotiation.cc:407] Connection header received
0619 03:01:07.410250 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.410253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.410316 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.410435 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:01:07.410804 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.411493 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.412831 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.413222 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.413307 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.413706 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:01:07.414138 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:01:07.414246 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.414310 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.414373 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 03:01:07.422209 8411 raft_consensus.cc:1261] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 1 FOLLOWER]: Refusing update from remote peer 35e0882c86114cdb8f053b18053695c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:07.422242 8277 raft_consensus.cc:1261] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 1 FOLLOWER]: Refusing update from remote peer 35e0882c86114cdb8f053b18053695c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:07.422998 8620 consensus_queue.cc:1035] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fbc8a315c34714bd06622000bb8ab1" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:07.423290 8621 consensus_queue.cc:1035] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0733654bd3bc4ecc869c9e03f77b92b6" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 36305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:07.435312 8624 mvcc.cc:205] Tried to move back new op lower bound from 7298408724152225792 to 7298408724054376448. Current Snapshot: MvccSnapshot[applied={T|T < 7298408724152225792}]
I20260619 03:01:07.448408 8133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:07.448693 8133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:07.452174 8133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2355bd69c36e43e08271150382424af6: Sending DeleteTablet for 3 replicas of tablet f496f280117a4beebadf213ff2b40208
I20260619 03:01:07.453219 8524 tablet_service.cc:1514] Processing DeleteTablet for tablet f496f280117a4beebadf213ff2b40208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:07 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:36536
I20260619 03:01:07.453796 8391 tablet_service.cc:1514] Processing DeleteTablet for tablet f496f280117a4beebadf213ff2b40208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:07 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56682
I20260619 03:01:07.453873 8257 tablet_service.cc:1514] Processing DeleteTablet for tablet f496f280117a4beebadf213ff2b40208 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:07 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:38346
I20260619 03:01:07.455394 8634 tablet_replica.cc:330] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: stopping tablet replica
I20260619 03:01:07.455401 8633 tablet_replica.cc:330] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: stopping tablet replica
I20260619 03:01:07.455651 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8194
I20260619 03:01:07.455778 8634 raft_consensus.cc:2229] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:07.455788 8633 raft_consensus.cc:2229] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:01:07.456234 8634 raft_consensus.cc:2258] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:07.456420 8633 raft_consensus.cc:2258] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:07.458436 8633 ts_tablet_manager.cc:1905] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:07.458528 8634 ts_tablet_manager.cc:1905] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:07.465277 8634 ts_tablet_manager.cc:1918] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:07.465435 8634 log.cc:1196] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/f496f280117a4beebadf213ff2b40208
I20260619 03:01:07.465975 8634 ts_tablet_manager.cc:1939] T f496f280117a4beebadf213ff2b40208 P 0733654bd3bc4ecc869c9e03f77b92b6: Deleting consensus metadata
I20260619 03:01:07.467053 8635 tablet_replica.cc:330] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: stopping tablet replica
I20260619 03:01:07.467278 8633 ts_tablet_manager.cc:1918] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:07.467469 8633 log.cc:1196] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/f496f280117a4beebadf213ff2b40208
I20260619 03:01:07.467490 8635 raft_consensus.cc:2229] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:07.467960 8635 raft_consensus.cc:2258] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:07.468058 8633 ts_tablet_manager.cc:1939] T f496f280117a4beebadf213ff2b40208 P 35e0882c86114cdb8f053b18053695c0: Deleting consensus metadata
I20260619 03:01:07.469971 8119 catalog_manager.cc:4841] TS 35e0882c86114cdb8f053b18053695c0 (127.22.230.3:40717): tablet f496f280117a4beebadf213ff2b40208 (table test-table [id=86a2ef49a8594d7c9b0b66d9cb6e414a]) successfully deleted
I20260619 03:01:07.470261 8635 ts_tablet_manager.cc:1905] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:07.471848 8118 catalog_manager.cc:4841] TS 0733654bd3bc4ecc869c9e03f77b92b6 (127.22.230.2:36305): tablet f496f280117a4beebadf213ff2b40208 (table test-table [id=86a2ef49a8594d7c9b0b66d9cb6e414a]) successfully deleted
I20260619 03:01:07.477352 8635 ts_tablet_manager.cc:1918] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:07.477499 8635 log.cc:1196] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/f496f280117a4beebadf213ff2b40208
I20260619 03:01:07.477975 8635 ts_tablet_manager.cc:1939] T f496f280117a4beebadf213ff2b40208 P 35fbc8a315c34714bd06622000bb8ab1: Deleting consensus metadata
I20260619 03:01:07.479538 8118 catalog_manager.cc:4841] TS 35fbc8a315c34714bd06622000bb8ab1 (127.22.230.1:33349): tablet f496f280117a4beebadf213ff2b40208 (table test-table [id=86a2ef49a8594d7c9b0b66d9cb6e414a]) successfully deleted
Jun 19 03:01:07 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 03:01:07.488521 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.458223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.22.230.1:33349)
0619 03:01:07.458380 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.458385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.472389 (+ 14004us) server_negotiation.cc:407] Connection header received
0619 03:01:07.472583 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.472589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.472678 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.472798 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:07.473955 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.474889 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.476179 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.476562 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.476719 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.481976 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:07.481989 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:07.482006 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:07.482045 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:07.485571 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.486360 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.486369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.486372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.486542 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.486940 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.486945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.486949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.487365 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:07.487641 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.487928 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.488141 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 03:01:07.489156 8317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:07.603633 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8194
I20260619 03:01:07.622043 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8328
Jun 19 03:01:07 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 03:01:07.637403 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.622864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.22.230.2:36305)
0619 03:01:07.623066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.623072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.623090 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:01:07.623252 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.623264 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.623337 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.623449 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:07.624635 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.625491 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.626920 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.627356 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.627501 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.631907 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:07.631920 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:07.631932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:07.631962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:07.634885 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.635646 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.635653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.635655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.635768 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.636164 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.636166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.636168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.636488 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:07.636698 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.636999 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.637142 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 03:01:07.638146 8451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:07.724020 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8328
I20260619 03:01:07.741837 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8461
Jun 19 03:01:07 dist-test-slave-2gms krb5kdc[8088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 03:01:07.759841 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:07.742643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.22.230.3:40717)
0619 03:01:07.742823 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:07.742830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:07.743061 (+ 231us) server_negotiation.cc:407] Connection header received
0619 03:01:07.743242 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:07.743245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:07.743330 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:07.743418 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:07.745195 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.745968 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.748221 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:07.748665 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:07.748763 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:07.753857 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:07.753869 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:07.753883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:07.753951 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:07.757053 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.757731 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.757738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.757741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.757867 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:07.758256 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:07.758260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:07.758262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:07.758723 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:07.759039 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:07.759285 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 03:01:07.759515 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 03:01:07.760630 8584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:07.878324 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8461
I20260619 03:01:07.898030 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8101
I20260619 03:01:07.899214 8163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:08.004694 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8101
2026-06-19T03:01:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T03:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:08Z Disabled control of system clock
I20260619 03:01:08.081943 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38731
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 03:01:08.383876 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:08.384167 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:08.384328 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:08.391228 8644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:08.391357 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:08.391448 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:08.391551 8644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:08.391633 8644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:08.400249 8644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:08.401417 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:08.403115 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:08.414369 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:08.414414 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:08.414446 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:08.415277 8644 server_base.cc:1034] running on GCE node
I20260619 03:01:08.415992 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:08.417272 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:08.418493 8644 hybrid_clock.cc:648] HybridClock initialized: now 1781838068418459 us; error 51 us; skew 500 ppm
I20260619 03:01:08.418848 8644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:08.421025 8644 webserver.cc:460] Webserver started at http://127.22.230.62:39071/ using document root <none> and password file <none>
I20260619 03:01:08.421674 8644 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:08.421802 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:08.422125 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:08.424922 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "e2d32cd6a43a4526ba3e32b43ccecc78"
format_stamp: "Formatted at 2026-06-19 03:01:08 on dist-test-slave-2gms"
server_key: "179e41fc0bbb4bcd26af0e026abac4e0"
server_key_iv: "74c9c960d5f75d9741677fe9222c0c78"
server_key_version: "encryptionkey@0"
I20260619 03:01:08.425709 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "e2d32cd6a43a4526ba3e32b43ccecc78"
format_stamp: "Formatted at 2026-06-19 03:01:08 on dist-test-slave-2gms"
server_key: "179e41fc0bbb4bcd26af0e026abac4e0"
server_key_iv: "74c9c960d5f75d9741677fe9222c0c78"
server_key_version: "encryptionkey@0"
I20260619 03:01:08.430639 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 03:01:08.433851 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:08.435179 8644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 03:01:08.435364 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "e2d32cd6a43a4526ba3e32b43ccecc78"
format_stamp: "Formatted at 2026-06-19 03:01:08 on dist-test-slave-2gms"
server_key: "179e41fc0bbb4bcd26af0e026abac4e0"
server_key_iv: "74c9c960d5f75d9741677fe9222c0c78"
server_key_version: "encryptionkey@0"
I20260619 03:01:08.435601 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:08.452894 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:08.453557 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:08.453770 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:08.471654 8644 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:38731
I20260619 03:01:08.471676 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:38731 every 8 connection(s)
I20260619 03:01:08.472842 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:08.476143 8710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:08.480198 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8644
I20260619 03:01:08.480367 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:08.480811 23448 external_mini_cluster.cc:1428] Setting key 3db46bd6219161e70c8524284090eeca
I20260619 03:01:08.486909 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78: Bootstrap starting.
I20260619 03:01:08.490185 8710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:08.491345 8710 log.cc:824] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:08.493757 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78: No bootstrap required, opened a new log
I20260619 03:01:08.497337 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:08.482505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.22.230.62:38731)
0619 03:01:08.483445 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:08.483455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:08.483495 (+ 40us) server_negotiation.cc:407] Connection header received
0619 03:01:08.485565 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:08.485586 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:08.485968 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:08.486263 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:08.486869 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:08.488177 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:08.490110 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:08.490867 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:08.490985 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:08.495194 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:08.495201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:08.495225 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:08.495751 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:08.495823 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:08.495846 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:08.495932 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:08.496179 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 03:01:08.496491 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":343,"threads_started":1}
I20260619 03:01:08.501286 8710 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38731 } }
I20260619 03:01:08.501628 8710 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:08.501729 8710 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d32cd6a43a4526ba3e32b43ccecc78, State: Initialized, Role: FOLLOWER
I20260619 03:01:08.502287 8710 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38731 } }
I20260619 03:01:08.502548 8710 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:08.502741 8710 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:08.502916 8710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:08.504967 8710 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38731 } }
I20260619 03:01:08.505429 8710 leader_election.cc:302] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2d32cd6a43a4526ba3e32b43ccecc78; no voters:
I20260619 03:01:08.505947 8710 leader_election.cc:288] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:08.506088 8715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:08.506809 8715 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [term 1 LEADER]: Becoming Leader. State: Replica: e2d32cd6a43a4526ba3e32b43ccecc78, State: Running, Role: LEADER
I20260619 03:01:08.507303 8715 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38731 } }
I20260619 03:01:08.507784 8710 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:08.509933 8717 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2d32cd6a43a4526ba3e32b43ccecc78. Latest consensus state: current_term: 1 leader_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38731 } } }
I20260619 03:01:08.510356 8717 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:08.510773 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2d32cd6a43a4526ba3e32b43ccecc78" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38731 } } }
I20260619 03:01:08.511125 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:08.512661 8720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:08.517618 8720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:08.526021 8720 catalog_manager.cc:1347] Generated new cluster ID: 5395db705aa74255b8e63aad38a12f73
I20260619 03:01:08.526134 8720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:08.548451 8720 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:08.549535 8720 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:08.559474 8720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78: Generated new TSK 0
I20260619 03:01:08.560061 8720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:08.566077 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38731
--builtin_ntp_servers=127.22.230.20:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 03:01:08.905117 8734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:08.905356 8734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:08.905437 8734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:08.912449 8734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:08.912528 8734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:08.912626 8734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:01:08.920892 8734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:08.921886 8734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:08.923563 8734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:08.935796 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:08.935756 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:08.935833 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:08.936506 8734 server_base.cc:1034] running on GCE node
I20260619 03:01:08.937049 8734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:08.937902 8734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:08.939172 8734 hybrid_clock.cc:648] HybridClock initialized: now 1781838068939112 us; error 44 us; skew 500 ppm
I20260619 03:01:08.939466 8734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:08.941550 8734 webserver.cc:460] Webserver started at http://127.22.230.1:34755/ using document root <none> and password file <none>
I20260619 03:01:08.942184 8734 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:08.942317 8734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:08.942652 8734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:08.945197 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "5ef9f33be4234a38bb5ecffbc7920b73"
format_stamp: "Formatted at 2026-06-19 03:01:08 on dist-test-slave-2gms"
server_key: "72ce8ba29b7be9cbee9c2ac59a4e7bc7"
server_key_iv: "d16d4ed96cbe69c5be4be1570379adde"
server_key_version: "encryptionkey@0"
I20260619 03:01:08.945947 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "5ef9f33be4234a38bb5ecffbc7920b73"
format_stamp: "Formatted at 2026-06-19 03:01:08 on dist-test-slave-2gms"
server_key: "72ce8ba29b7be9cbee9c2ac59a4e7bc7"
server_key_iv: "d16d4ed96cbe69c5be4be1570379adde"
server_key_version: "encryptionkey@0"
I20260619 03:01:08.950608 8734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 03:01:08.953629 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:08.954802 8734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260619 03:01:08.954972 8734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "5ef9f33be4234a38bb5ecffbc7920b73"
format_stamp: "Formatted at 2026-06-19 03:01:08 on dist-test-slave-2gms"
server_key: "72ce8ba29b7be9cbee9c2ac59a4e7bc7"
server_key_iv: "d16d4ed96cbe69c5be4be1570379adde"
server_key_version: "encryptionkey@0"
I20260619 03:01:08.955165 8734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:08.970873 8734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:08.971529 8734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:08.971807 8734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:08.972721 8734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:08.973996 8734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:08.974112 8734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:08.974236 8734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:08.974320 8734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.004940 8734 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:41847
I20260619 03:01:09.004985 8860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:41847 every 8 connection(s)
I20260619 03:01:09.006204 8734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:01:09.013427 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8734
I20260619 03:01:09.013621 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:01:09.014225 23448 external_mini_cluster.cc:1428] Setting key 58e4a188b151c3e1c4b600efb06451ed
I20260619 03:01:09.017009 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38731
--builtin_ntp_servers=127.22.230.20:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 03:01:09.019832 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.008787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:52219 (local address 127.22.230.62:38731)
0619 03:01:09.009284 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.009289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.010373 (+ 1084us) server_negotiation.cc:407] Connection header received
0619 03:01:09.012166 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.012172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.012263 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.012398 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:09.013564 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.014666 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.016449 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.017093 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.017240 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.019051 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:09.019058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:09.019063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:09.019083 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:09.019146 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:09.019176 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:09.019289 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.019534 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.019607 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
I20260619 03:01:09.021266 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.009432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38731 (local address 127.22.230.1:52219)
0619 03:01:09.010232 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:09.010271 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:09.011897 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:09.012622 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:09.012629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:09.012714 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:09.013343 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.013360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.014835 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.014840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.016313 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.016325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.017546 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.017556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.017887 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.018662 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:09.018667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:09.018717 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:09.018840 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:09.019299 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:09.019312 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:09.019329 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 03:01:09.019518 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 03:01:09.019914 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":235,"threads_started":1}
I20260619 03:01:09.022789 8861 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38731
I20260619 03:01:09.023070 8861 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:09.023756 8861 heartbeater.cc:511] Master 127.22.230.62:38731 requested a full tablet report, sending...
I20260619 03:01:09.025815 8675 ts_manager.cc:194] Registered new tserver with Master: 5ef9f33be4234a38bb5ecffbc7920b73 (127.22.230.1:41847)
I20260619 03:01:09.027771 8675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.1:52219
W20260619 03:01:09.326939 8865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:09.327169 8865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:09.327302 8865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:09.333971 8865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:09.334048 8865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:09.334213 8865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:01:09.343029 8865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:09.344074 8865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:09.345738 8865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:09.357986 8870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:09.358040 8873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:09.358067 8871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:09.359126 8865 server_base.cc:1034] running on GCE node
I20260619 03:01:09.359666 8865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:09.360484 8865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:09.361696 8865 hybrid_clock.cc:648] HybridClock initialized: now 1781838069361657 us; error 53 us; skew 500 ppm
I20260619 03:01:09.361992 8865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:09.363890 8865 webserver.cc:460] Webserver started at http://127.22.230.2:44473/ using document root <none> and password file <none>
I20260619 03:01:09.364493 8865 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:09.364624 8865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:09.364920 8865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:09.367620 8865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "a18641c933a74750bf54c9d7c135983b"
format_stamp: "Formatted at 2026-06-19 03:01:09 on dist-test-slave-2gms"
server_key: "38bec4098b50a7409e620507cc6de28f"
server_key_iv: "a570b7995de82a41c938637e3e51786f"
server_key_version: "encryptionkey@0"
I20260619 03:01:09.368368 8865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "a18641c933a74750bf54c9d7c135983b"
format_stamp: "Formatted at 2026-06-19 03:01:09 on dist-test-slave-2gms"
server_key: "38bec4098b50a7409e620507cc6de28f"
server_key_iv: "a570b7995de82a41c938637e3e51786f"
server_key_version: "encryptionkey@0"
I20260619 03:01:09.372913 8865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 03:01:09.376039 8879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.377205 8865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:01:09.377357 8865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "a18641c933a74750bf54c9d7c135983b"
format_stamp: "Formatted at 2026-06-19 03:01:09 on dist-test-slave-2gms"
server_key: "38bec4098b50a7409e620507cc6de28f"
server_key_iv: "a570b7995de82a41c938637e3e51786f"
server_key_version: "encryptionkey@0"
I20260619 03:01:09.377555 8865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:09.389578 8865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:09.390219 8865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:09.390494 8865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:09.391350 8865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:09.392638 8865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:09.392752 8865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.392870 8865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:09.392956 8865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.424278 8865 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:43553
I20260619 03:01:09.424337 8991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:43553 every 8 connection(s)
I20260619 03:01:09.425482 8865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:01:09.433101 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8865
I20260619 03:01:09.433276 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:01:09.433792 23448 external_mini_cluster.cc:1428] Setting key 1294ee23a17a8d6ab4482f2de647c8a5
I20260619 03:01:09.436797 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38731
--builtin_ntp_servers=127.22.230.20:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 03:01:09.437616 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.427810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:46101 (local address 127.22.230.62:38731)
0619 03:01:09.428030 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.428036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.429324 (+ 1288us) server_negotiation.cc:407] Connection header received
0619 03:01:09.431027 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.431031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.431093 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.431177 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:09.432099 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.432779 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.434900 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.435361 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.435494 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.436870 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:09.436896 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:09.436902 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:09.436922 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:09.436968 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:09.436988 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:09.437069 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.437311 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.437394 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 03:01:09.438948 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.428356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38731 (local address 127.22.230.2:46101)
0619 03:01:09.429181 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:09.429216 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:09.430799 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:09.431362 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:09.431370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:09.431448 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:09.431966 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.431982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.432933 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.432938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.434737 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.434755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.435580 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.435587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.435879 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.436554 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:09.436558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:09.436620 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:09.436703 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:09.437080 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:09.437091 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:09.437101 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 03:01:09.437293 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 03:01:09.437679 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":249,"threads_started":1}
I20260619 03:01:09.440960 8992 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38731
I20260619 03:01:09.441259 8992 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:09.442068 8992 heartbeater.cc:511] Master 127.22.230.62:38731 requested a full tablet report, sending...
I20260619 03:01:09.443384 8675 ts_manager.cc:194] Registered new tserver with Master: a18641c933a74750bf54c9d7c135983b (127.22.230.2:43553)
I20260619 03:01:09.444180 8675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.2:46101
W20260619 03:01:09.742102 8995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:09.742321 8995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:09.742475 8995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:09.749079 8995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:09.749154 8995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:09.749249 8995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:01:09.757448 8995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:38731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:09.758536 8995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:09.760099 8995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:09.772068 9000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:09.772137 9001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:09.772100 9003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:09.772877 8995 server_base.cc:1034] running on GCE node
I20260619 03:01:09.773345 8995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:09.774117 8995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:09.775300 8995 hybrid_clock.cc:648] HybridClock initialized: now 1781838069775247 us; error 45 us; skew 500 ppm
I20260619 03:01:09.775591 8995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:09.777642 8995 webserver.cc:460] Webserver started at http://127.22.230.3:36055/ using document root <none> and password file <none>
I20260619 03:01:09.778252 8995 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:09.778357 8995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:09.778704 8995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:09.781322 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7"
format_stamp: "Formatted at 2026-06-19 03:01:09 on dist-test-slave-2gms"
server_key: "2e3c5d9f6d7c194f617e38239df9fd01"
server_key_iv: "90a2fa9764e1dd4d69b2857fcce48e97"
server_key_version: "encryptionkey@0"
I20260619 03:01:09.782146 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7"
format_stamp: "Formatted at 2026-06-19 03:01:09 on dist-test-slave-2gms"
server_key: "2e3c5d9f6d7c194f617e38239df9fd01"
server_key_iv: "90a2fa9764e1dd4d69b2857fcce48e97"
server_key_version: "encryptionkey@0"
I20260619 03:01:09.786753 8995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260619 03:01:09.789813 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.791023 8995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:09.791183 8995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7"
format_stamp: "Formatted at 2026-06-19 03:01:09 on dist-test-slave-2gms"
server_key: "2e3c5d9f6d7c194f617e38239df9fd01"
server_key_iv: "90a2fa9764e1dd4d69b2857fcce48e97"
server_key_version: "encryptionkey@0"
I20260619 03:01:09.791383 8995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:09.802556 8995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:09.803287 8995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:09.803581 8995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:09.804373 8995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:09.805656 8995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:09.805776 8995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.805904 8995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:09.805994 8995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:09.837280 8995 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:34165
I20260619 03:01:09.837353 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:34165 every 8 connection(s)
I20260619 03:01:09.838418 8995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:01:09.842342 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 8995
I20260619 03:01:09.842494 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:01:09.842939 23448 external_mini_cluster.cc:1428] Setting key 041677b5475633654b541209b7d3d72b
I20260619 03:01:09.849867 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.840677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:52609 (local address 127.22.230.62:38731)
0619 03:01:09.840828 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.840832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.842036 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 03:01:09.843740 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.843746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.843818 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.843909 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:09.845028 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.845999 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.847634 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.848056 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.848154 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.849230 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:09.849234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:09.849237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:09.849253 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:09.849283 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:09.849295 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:09.849375 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.849546 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.849616 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 03:01:09.850744 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.841148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:38731 (local address 127.22.230.3:52609)
0619 03:01:09.841892 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:09.841935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:09.843481 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:09.844147 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:09.844156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:09.844251 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:09.844879 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.844894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.846128 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.846132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.847396 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.847408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.848226 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.848230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.848401 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.848989 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:09.848992 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:09.849023 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:09.849084 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:09.849385 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:09.849391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:09.849396 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 03:01:09.849520 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 03:01:09.849748 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":232,"threads_started":1}
I20260619 03:01:09.851881 9122 heartbeater.cc:347] Connected to a master server at 127.22.230.62:38731
I20260619 03:01:09.852145 9122 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:09.852725 9122 heartbeater.cc:511] Master 127.22.230.62:38731 requested a full tablet report, sending...
I20260619 03:01:09.853896 8675 ts_manager.cc:194] Registered new tserver with Master: 160e17a6b40b4d4eadaa6b5f4bae4df7 (127.22.230.3:34165)
I20260619 03:01:09.854753 8675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.3:52609
I20260619 03:01:09.857517 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:01:09.866551 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.861313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.22.230.62:38731)
0619 03:01:09.861443 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.861447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.861923 (+ 476us) server_negotiation.cc:407] Connection header received
0619 03:01:09.862036 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.862039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.862100 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.862175 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:09.862651 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.863287 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.864738 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.865123 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.865206 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.866052 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:09.866056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:09.866059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:09.866074 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:09.866099 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:09.866110 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:09.866175 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.866294 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.866363 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 03:01:09.869691 8675 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:01:09.876142 8675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:01:09.915042 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.887177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:34165 (local address 127.0.0.1:44510)
0619 03:01:09.907266 (+ 20089us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:09.907279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:09.907414 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:09.908222 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:09.908230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:09.908270 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:09.908672 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.908683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.910983 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.910990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.912534 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.912546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.914091 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.914099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.914275 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.914320 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 03:01:09.914460 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 03:01:09.914685 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20001,"thread_start_us":20806,"threads_started":1}
I20260619 03:01:09.915083 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.887537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.22.230.3:34165)
0619 03:01:09.904280 (+ 16743us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.904288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.907376 (+ 3088us) server_negotiation.cc:407] Connection header received
0619 03:01:09.907640 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.907647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.907945 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.908122 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:09.908893 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.910847 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.912711 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.913905 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.914035 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.914287 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.914536 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.914703 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16586,"thread_start_us":16743,"threads_started":1}
I20260619 03:01:09.915890 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.889089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:43553 (local address 127.0.0.1:49542)
0619 03:01:09.908211 (+ 19122us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:09.908226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:09.908336 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:09.909032 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:09.909036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:09.909059 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:09.909386 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.909394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.911051 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.911056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.913219 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.913229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.915135 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.915140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.915339 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.915372 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 03:01:09.915463 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 03:01:09.915604 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19007,"thread_start_us":17972,"threads_started":1}
I20260619 03:01:09.916498 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.887183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.22.230.1:41847)
0619 03:01:09.907814 (+ 20631us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.907821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.907847 (+ 26us) server_negotiation.cc:407] Connection header received
0619 03:01:09.907945 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.907952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.908179 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.908405 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:09.909058 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.910470 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.913349 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.914835 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.914970 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.915213 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.915992 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.916148 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20488,"thread_start_us":20383,"threads_started":1}
I20260619 03:01:09.916581 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.889812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.22.230.2:43553)
0619 03:01:09.906951 (+ 17139us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:09.906957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:09.908480 (+ 1523us) server_negotiation.cc:407] Connection header received
0619 03:01:09.908583 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:09.908592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:09.908790 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:09.909016 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:09.909537 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.910891 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.913406 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.914655 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.914766 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.914986 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:09.915693 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 03:01:09.915831 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17014,"thread_start_us":16932,"threads_started":1}
I20260619 03:01:09.917049 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:09.887039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:60946)
0619 03:01:09.902311 (+ 15272us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:09.902368 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:09.902539 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:09.908406 (+ 5867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:09.908410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:09.908437 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:09.908852 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.908865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.910644 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.910649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.913157 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:09.913177 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:09.915336 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:09.915343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:09.915540 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:09.915570 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 03:01:09.915656 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 03:01:09.915772 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14982,"mutex_wait_us":118,"thread_start_us":15701,"threads_started":1}
I20260619 03:01:09.918882 9057 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9856419eba4e1ab83fbc86cb06adda (DEFAULT_TABLE table=test-table [id=3f34b16442c8410c93de00ab53e97983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:09.920011 9057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9856419eba4e1ab83fbc86cb06adda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:09.919772 8927 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9856419eba4e1ab83fbc86cb06adda (DEFAULT_TABLE table=test-table [id=3f34b16442c8410c93de00ab53e97983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:09.921031 8927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9856419eba4e1ab83fbc86cb06adda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:09.920595 8796 tablet_service.cc:1467] Processing CreateTablet for tablet 9c9856419eba4e1ab83fbc86cb06adda (DEFAULT_TABLE table=test-table [id=3f34b16442c8410c93de00ab53e97983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:09.921738 8796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c9856419eba4e1ab83fbc86cb06adda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:09.930153 9140 tablet_bootstrap.cc:492] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Bootstrap starting.
I20260619 03:01:09.933174 9140 tablet_bootstrap.cc:654] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:09.934265 9140 log.cc:824] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:09.934482 9141 tablet_bootstrap.cc:492] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Bootstrap starting.
I20260619 03:01:09.935835 9142 tablet_bootstrap.cc:492] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Bootstrap starting.
I20260619 03:01:09.938752 9141 tablet_bootstrap.cc:654] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:09.938789 9142 tablet_bootstrap.cc:654] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:09.939919 9141 log.cc:824] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:09.939961 9142 log.cc:824] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:09.941248 9140 tablet_bootstrap.cc:492] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: No bootstrap required, opened a new log
I20260619 03:01:09.941514 9140 ts_tablet_manager.cc:1397] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260619 03:01:09.942534 9142 tablet_bootstrap.cc:492] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: No bootstrap required, opened a new log
I20260619 03:01:09.942662 9141 tablet_bootstrap.cc:492] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: No bootstrap required, opened a new log
I20260619 03:01:09.942844 9142 ts_tablet_manager.cc:1397] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260619 03:01:09.942916 9141 ts_tablet_manager.cc:1397] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260619 03:01:09.949657 9142 raft_consensus.cc:348] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:09.950078 9142 raft_consensus.cc:374] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:09.950234 9142 raft_consensus.cc:729] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef9f33be4234a38bb5ecffbc7920b73, State: Initialized, Role: FOLLOWER
I20260619 03:01:09.950852 9142 consensus_queue.cc:260] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:09.951686 9140 raft_consensus.cc:348] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:09.951941 8861 heartbeater.cc:503] Master 127.22.230.62:38731 was elected leader, sending a full tablet report...
I20260619 03:01:09.952061 9140 raft_consensus.cc:374] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:09.952143 9140 raft_consensus.cc:729] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 160e17a6b40b4d4eadaa6b5f4bae4df7, State: Initialized, Role: FOLLOWER
I20260619 03:01:09.952719 9142 ts_tablet_manager.cc:1428] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 03:01:09.952667 9140 consensus_queue.cc:260] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:09.952781 9141 raft_consensus.cc:348] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:09.953197 9141 raft_consensus.cc:374] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:09.953294 9141 raft_consensus.cc:729] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18641c933a74750bf54c9d7c135983b, State: Initialized, Role: FOLLOWER
I20260619 03:01:09.953650 9122 heartbeater.cc:503] Master 127.22.230.62:38731 was elected leader, sending a full tablet report...
I20260619 03:01:09.953866 9141 consensus_queue.cc:260] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:09.954301 9140 ts_tablet_manager.cc:1428] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 03:01:09.955360 8992 heartbeater.cc:503] Master 127.22.230.62:38731 was elected leader, sending a full tablet report...
I20260619 03:01:09.955926 9141 ts_tablet_manager.cc:1428] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260619 03:01:10.008632 8862 tablet.cc:2369] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:01:10.008631 9147 raft_consensus.cc:482] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:10.008939 9147 raft_consensus.cc:504] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:10.010528 9147 leader_election.cc:288] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ef9f33be4234a38bb5ecffbc7920b73 (127.22.230.1:41847), a18641c933a74750bf54c9d7c135983b (127.22.230.2:43553)
I20260619 03:01:10.016927 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.010876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.22.230.3:57225)
0619 03:01:10.011167 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.011188 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:10.011305 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:10.011970 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:10.011974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:10.012003 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:10.012197 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:10.012208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.013489 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.013494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:10.015091 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:10.015102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.016278 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.016285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:10.016459 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.016489 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 03:01:10.016567 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 03:01:10.016680 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260619 03:01:10.016948 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.011362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:57225 (local address 127.22.230.1:41847)
0619 03:01:10.011593 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:10.011599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:10.011617 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:01:10.011680 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:10.011685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:10.011784 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:10.011906 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:10.012358 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.013352 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.015237 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.016107 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.016281 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.016343 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:10.016605 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 03:01:10.016717 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 03:01:10.017793 8816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9856419eba4e1ab83fbc86cb06adda" candidate_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" is_pre_election: true
I20260619 03:01:10.018095 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.011122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:43553 (local address 127.22.230.3:49379)
0619 03:01:10.011875 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.011893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:10.012068 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:10.012392 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:10.012396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:10.012417 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:10.012703 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:10.012709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.013814 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.013819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:10.015844 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:10.015857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.017467 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.017472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:10.017620 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.017647 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 03:01:10.017730 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 03:01:10.017832 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":446,"threads_started":1}
I20260619 03:01:10.018157 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.010943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:49379 (local address 127.22.230.2:43553)
0619 03:01:10.011124 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:10.011129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:10.011969 (+ 840us) server_negotiation.cc:407] Connection header received
0619 03:01:10.012214 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:10.012219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:10.012301 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:10.012382 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:10.012837 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.013645 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.016040 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.017081 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.017232 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.017300 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:10.017775 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 03:01:10.017895 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 03:01:10.018330 8816 raft_consensus.cc:2454] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 160e17a6b40b4d4eadaa6b5f4bae4df7 in term 0.
I20260619 03:01:10.019042 9011 leader_election.cc:302] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 160e17a6b40b4d4eadaa6b5f4bae4df7, 5ef9f33be4234a38bb5ecffbc7920b73; no voters:
I20260619 03:01:10.019076 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9856419eba4e1ab83fbc86cb06adda" candidate_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18641c933a74750bf54c9d7c135983b" is_pre_election: true
I20260619 03:01:10.019408 9147 raft_consensus.cc:2781] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:01:10.019542 9147 raft_consensus.cc:482] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:01:10.019493 8947 raft_consensus.cc:2454] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 160e17a6b40b4d4eadaa6b5f4bae4df7 in term 0.
I20260619 03:01:10.019650 9147 raft_consensus.cc:3037] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:10.021771 9147 raft_consensus.cc:504] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:10.022377 9147 leader_election.cc:288] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [CANDIDATE]: Term 1 election: Requested vote from peers 5ef9f33be4234a38bb5ecffbc7920b73 (127.22.230.1:41847), a18641c933a74750bf54c9d7c135983b (127.22.230.2:43553)
I20260619 03:01:10.022686 8816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9856419eba4e1ab83fbc86cb06adda" candidate_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef9f33be4234a38bb5ecffbc7920b73"
I20260619 03:01:10.022717 8947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c9856419eba4e1ab83fbc86cb06adda" candidate_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18641c933a74750bf54c9d7c135983b"
I20260619 03:01:10.022847 8816 raft_consensus.cc:3037] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:10.022879 8947 raft_consensus.cc:3037] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:10.024917 8816 raft_consensus.cc:2454] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 160e17a6b40b4d4eadaa6b5f4bae4df7 in term 1.
I20260619 03:01:10.025086 8947 raft_consensus.cc:2454] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 160e17a6b40b4d4eadaa6b5f4bae4df7 in term 1.
I20260619 03:01:10.025293 9011 leader_election.cc:302] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 160e17a6b40b4d4eadaa6b5f4bae4df7, 5ef9f33be4234a38bb5ecffbc7920b73; no voters:
I20260619 03:01:10.025571 9147 raft_consensus.cc:2781] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:10.026012 9147 raft_consensus.cc:686] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 1 LEADER]: Becoming Leader. State: Replica: 160e17a6b40b4d4eadaa6b5f4bae4df7, State: Running, Role: LEADER
I20260619 03:01:10.026427 9147 consensus_queue.cc:237] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } }
I20260619 03:01:10.030272 8674 catalog_manager.cc:5495] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 160e17a6b40b4d4eadaa6b5f4bae4df7 (127.22.230.3). New cstate: current_term: 1 leader_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "160e17a6b40b4d4eadaa6b5f4bae4df7" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 } health_report { overall_health: UNKNOWN } } }
I20260619 03:01:10.048841 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.043565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.22.230.3:34165)
0619 03:01:10.043718 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:10.043725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:10.043743 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:01:10.043959 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:10.043963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:10.044023 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:10.044153 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:01:10.044503 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.045226 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.046804 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.047267 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.047405 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.047869 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:01:10.048300 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:01:10.048465 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:10.048547 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 03:01:10.048614 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 03:01:10.056089 8947 raft_consensus.cc:1261] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 1 FOLLOWER]: Refusing update from remote peer 160e17a6b40b4d4eadaa6b5f4bae4df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:10.056089 8816 raft_consensus.cc:1261] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 1 FOLLOWER]: Refusing update from remote peer 160e17a6b40b4d4eadaa6b5f4bae4df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:10.057039 9150 consensus_queue.cc:1035] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef9f33be4234a38bb5ecffbc7920b73" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:10.057350 9147 consensus_queue.cc:1035] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18641c933a74750bf54c9d7c135983b" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:10.073637 9153 mvcc.cc:205] Tried to move back new op lower bound from 7298408734939639808 to 7298408734831894528. Current Snapshot: MvccSnapshot[applied={T|T < 7298408734939639808}]
I20260619 03:01:10.073968 9154 mvcc.cc:205] Tried to move back new op lower bound from 7298408734939639808 to 7298408734831894528. Current Snapshot: MvccSnapshot[applied={T|T < 7298408734939639808}]
I20260619 03:01:10.077234 9155 mvcc.cc:205] Tried to move back new op lower bound from 7298408734939639808 to 7298408734831894528. Current Snapshot: MvccSnapshot[applied={T|T < 7298408734939639808}]
I20260619 03:01:10.085249 8674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:10.085495 8674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:10.088876 8674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2d32cd6a43a4526ba3e32b43ccecc78: Sending DeleteTablet for 3 replicas of tablet 9c9856419eba4e1ab83fbc86cb06adda
I20260619 03:01:10.089797 8796 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9856419eba4e1ab83fbc86cb06adda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:10 UTC) from {username='slave'} at 127.0.0.1:60946
I20260619 03:01:10.090281 8927 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9856419eba4e1ab83fbc86cb06adda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:10 UTC) from {username='slave'} at 127.0.0.1:49542
I20260619 03:01:10.090619 9162 tablet_replica.cc:330] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: stopping tablet replica
I20260619 03:01:10.090672 9057 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c9856419eba4e1ab83fbc86cb06adda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:10 UTC) from {username='slave'} at 127.0.0.1:44510
I20260619 03:01:10.091012 9162 raft_consensus.cc:2229] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:10.091545 9162 raft_consensus.cc:2258] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:10.092424 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8734
I20260619 03:01:10.095141 9164 tablet_replica.cc:330] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: stopping tablet replica
I20260619 03:01:10.095702 9164 raft_consensus.cc:2229] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:01:10.096467 9164 raft_consensus.cc:2258] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:10.097261 9162 ts_tablet_manager.cc:1905] T 9c9856419eba4e1ab83fbc86cb06adda P 5ef9f33be4234a38bb5ecffbc7920b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:10.098716 9164 ts_tablet_manager.cc:1905] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:10.105540 9163 tablet_replica.cc:330] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: stopping tablet replica
I20260619 03:01:10.105989 9163 raft_consensus.cc:2229] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:10.106467 9163 raft_consensus.cc:2258] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:10.106943 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.093504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.22.230.1:41847)
0619 03:01:10.093699 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:10.093707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:10.099341 (+ 5634us) server_negotiation.cc:407] Connection header received
0619 03:01:10.099560 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:10.099565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:10.099658 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:10.099760 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:10.100309 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.101356 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.103234 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.103844 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.103935 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.105317 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:10.105328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:10.105344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:10.105996 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:10.106051 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:10.106074 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:10.106153 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:10.106405 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 03:01:10.106520 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:01:10.107800 9164 ts_tablet_manager.cc:1918] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:10.107846 8856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:10.108021 9164 log.cc:1196] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/9c9856419eba4e1ab83fbc86cb06adda
I20260619 03:01:10.108614 9163 ts_tablet_manager.cc:1905] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:10.108745 9164 ts_tablet_manager.cc:1939] T 9c9856419eba4e1ab83fbc86cb06adda P 160e17a6b40b4d4eadaa6b5f4bae4df7: Deleting consensus metadata
I20260619 03:01:10.110419 8662 catalog_manager.cc:4841] TS 160e17a6b40b4d4eadaa6b5f4bae4df7 (127.22.230.3:34165): tablet 9c9856419eba4e1ab83fbc86cb06adda (table test-table [id=3f34b16442c8410c93de00ab53e97983]) successfully deleted
I20260619 03:01:10.116092 9163 ts_tablet_manager.cc:1918] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:10.116237 9163 log.cc:1196] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/9c9856419eba4e1ab83fbc86cb06adda
I20260619 03:01:10.116727 9163 ts_tablet_manager.cc:1939] T 9c9856419eba4e1ab83fbc86cb06adda P a18641c933a74750bf54c9d7c135983b: Deleting consensus metadata
I20260619 03:01:10.118350 8661 catalog_manager.cc:4841] TS a18641c933a74750bf54c9d7c135983b (127.22.230.2:43553): tablet 9c9856419eba4e1ab83fbc86cb06adda (table test-table [id=3f34b16442c8410c93de00ab53e97983]) successfully deleted
I20260619 03:01:10.200143 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8734
W20260619 03:01:10.221504 8660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.230.1:41847 (error 108)
I20260619 03:01:10.222194 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8865
I20260619 03:01:10.223666 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.223111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:60964)
0619 03:01:10.223357 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.223499 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260619 03:01:10.223953 8660 catalog_manager.cc:4584] TS 5ef9f33be4234a38bb5ecffbc7920b73 (127.22.230.1:41847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c9856419eba4e1ab83fbc86cb06adda: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
I20260619 03:01:10.229846 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.223001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.22.230.2:43553)
0619 03:01:10.223194 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:10.223200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:10.223264 (+ 64us) server_negotiation.cc:407] Connection header received
0619 03:01:10.223463 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:10.223469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:10.223583 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:10.223697 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:10.224167 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.225298 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.226593 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.227251 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.227372 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.228349 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:10.228360 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:10.228377 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:10.229108 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:10.229164 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:10.229187 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:10.229295 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:10.229392 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 03:01:10.229498 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 03:01:10.230558 8987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:10.277683 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.277173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:60972)
0619 03:01:10.277398 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.277505 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 03:01:10.278780 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.278369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:60974)
0619 03:01:10.278526 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.278637 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 03:01:10.314987 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8865
I20260619 03:01:10.339391 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.338791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:60984)
0619 03:01:10.339116 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.339211 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 03:01:10.339442 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8995
I20260619 03:01:10.340581 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.340105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:32768)
0619 03:01:10.340365 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.340456 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260619 03:01:10.347033 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.340253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.22.230.3:34165)
0619 03:01:10.340413 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:10.340421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:10.340472 (+ 51us) server_negotiation.cc:407] Connection header received
0619 03:01:10.340691 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:10.340699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:10.340790 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:10.340917 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:10.341345 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.342189 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.344061 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:10.344449 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:10.344539 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:10.345626 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:10.345634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:10.345646 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:10.346206 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:10.346249 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:10.346280 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:10.346373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:10.346547 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 03:01:10.346672 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 03:01:10.347760 9117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:10.420828 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.420372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:32782)
0619 03:01:10.420572 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.420668 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 03:01:10.421810 9134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:10.421402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41847 (local address 127.0.0.1:32796)
0619 03:01:10.421594 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:10.421668 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 03:01:10.461334 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8995
I20260619 03:01:10.482070 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 8644
I20260619 03:01:10.483150 8705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:10.585791 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 8644
2026-06-19T03:01:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2580 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:10 dist-test-slave-2gms krb5kdc[9173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:10 dist-test-slave-2gms krb5kdc[9173](info): set up 2 sockets
Jun 19 03:01:10 dist-test-slave-2gms krb5kdc[9173](info): commencing operation
krb5kdc: starting...
W20260619 03:01:11.650367 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:11 dist-test-slave-2gms krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:11.867625 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36089
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:36089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:12.171903 9186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:12.172194 9186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:12.172341 9186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:12.179248 9186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:12.179373 9186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:12.179476 9186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:12.179579 9186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:12.179664 9186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:12.188938 9186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:36089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:12.190099 9186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:12.191310 9186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:12.202203 9191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:12.202212 9192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:12.202226 9194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:12.202831 9186 server_base.cc:1034] running on GCE node
I20260619 03:01:12.203565 9186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:12.204671 9186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:12.205860 9186 hybrid_clock.cc:648] HybridClock initialized: now 1781838072205822 us; error 30 us; skew 500 ppm
I20260619 03:01:12.206137 9186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:12 dist-test-slave-2gms krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838072, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:12.208984 9186 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:12.210177 9186 webserver.cc:460] Webserver started at http://127.22.230.62:42595/ using document root <none> and password file <none>
I20260619 03:01:12.210816 9186 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:12.210912 9186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:12.211135 9186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:12.213711 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "68cc0592439b492faccf443852c398c7"
format_stamp: "Formatted at 2026-06-19 03:01:12 on dist-test-slave-2gms"
server_key: "119d8eb34af1875d10e80efdad48c940"
server_key_iv: "b164cfa1dfb4fbe707d7e1b4ecf2b57c"
server_key_version: "encryptionkey@0"
I20260619 03:01:12.214354 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "68cc0592439b492faccf443852c398c7"
format_stamp: "Formatted at 2026-06-19 03:01:12 on dist-test-slave-2gms"
server_key: "119d8eb34af1875d10e80efdad48c940"
server_key_iv: "b164cfa1dfb4fbe707d7e1b4ecf2b57c"
server_key_version: "encryptionkey@0"
I20260619 03:01:12.219089 9186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:01:12.222119 9201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:12.223296 9186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:12.223481 9186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "68cc0592439b492faccf443852c398c7"
format_stamp: "Formatted at 2026-06-19 03:01:12 on dist-test-slave-2gms"
server_key: "119d8eb34af1875d10e80efdad48c940"
server_key_iv: "b164cfa1dfb4fbe707d7e1b4ecf2b57c"
server_key_version: "encryptionkey@0"
I20260619 03:01:12.223654 9186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:12.244170 9186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:12.247764 9186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:12.248026 9186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:12.264604 9186 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:36089
I20260619 03:01:12.264627 9252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:36089 every 8 connection(s)
I20260619 03:01:12.265775 9186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:12.267576 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9186
I20260619 03:01:12.267746 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:12.268281 23448 external_mini_cluster.cc:1428] Setting key 3bb7a49960dbad773ac224d78762e36a
I20260619 03:01:12.269737 9253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 03:01:12 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:12.282133 9253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7: Bootstrap starting.
I20260619 03:01:12.285699 9253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:12.286983 9253 log.cc:824] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:12.290210 9253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7: No bootstrap required, opened a new log
I20260619 03:01:12.291255 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:12.269967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.22.230.62:36089)
0619 03:01:12.270682 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:12.270696 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:12.270739 (+ 43us) server_negotiation.cc:407] Connection header received
0619 03:01:12.272603 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:12.272626 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:12.273106 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:12.273497 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:12.274689 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:12.276717 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:12.278019 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:12.279041 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:12.279227 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:12.283164 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:12.283173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:12.283188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:12.283227 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:12.286877 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:12.287606 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:12.287620 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:12.287628 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:12.287784 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:12.288230 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:12.288234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:12.288237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:12.288858 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:12.289170 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:12.289613 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 03:01:12.290014 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":278,"threads_started":1}
I20260619 03:01:12.298237 9253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cc0592439b492faccf443852c398c7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36089 } }
I20260619 03:01:12.298516 9253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:12.298637 9253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cc0592439b492faccf443852c398c7, State: Initialized, Role: FOLLOWER
I20260619 03:01:12.299140 9253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cc0592439b492faccf443852c398c7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36089 } }
I20260619 03:01:12.299328 9253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:12.299420 9253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:12.299535 9253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:12.301388 9253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cc0592439b492faccf443852c398c7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36089 } }
I20260619 03:01:12.301760 9253 leader_election.cc:302] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68cc0592439b492faccf443852c398c7; no voters:
I20260619 03:01:12.302233 9253 leader_election.cc:288] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:12.302371 9258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:12.303061 9258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [term 1 LEADER]: Becoming Leader. State: Replica: 68cc0592439b492faccf443852c398c7, State: Running, Role: LEADER
I20260619 03:01:12.303668 9258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cc0592439b492faccf443852c398c7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36089 } }
I20260619 03:01:12.304327 9253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:12.306241 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68cc0592439b492faccf443852c398c7. Latest consensus state: current_term: 1 leader_uuid: "68cc0592439b492faccf443852c398c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cc0592439b492faccf443852c398c7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36089 } } }
I20260619 03:01:12.306617 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:12.307425 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68cc0592439b492faccf443852c398c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68cc0592439b492faccf443852c398c7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36089 } } }
I20260619 03:01:12.307715 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:12.308240 9265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:12.312566 9265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:12.320500 9265 catalog_manager.cc:1347] Generated new cluster ID: 034e570467124fa0b7c08ec7204d29ba
I20260619 03:01:12.320595 9265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:12.344911 9265 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:12.346467 9265 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:12.359928 9265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7: Generated new TSK 0
I20260619 03:01:12.360787 9265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 03:01:12.427865 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36089
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:12.733976 9279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:12.734196 9279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:12.734277 9279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:12.740983 9279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:12.741067 9279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:12.741199 9279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:01:12.749910 9279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:12.751067 9279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:12.752234 9279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:12.764336 9285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:12.764303 9284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:12.764367 9287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:12.765208 9279 server_base.cc:1034] running on GCE node
I20260619 03:01:12.765797 9279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:12.766760 9279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:12.767989 9279 hybrid_clock.cc:648] HybridClock initialized: now 1781838072767961 us; error 40 us; skew 500 ppm
I20260619 03:01:12.768270 9279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:12 dist-test-slave-2gms krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838072, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:12.771258 9279 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 03:01:12.772642 9279 webserver.cc:460] Webserver started at http://127.22.230.1:40213/ using document root <none> and password file <none>
I20260619 03:01:12.773221 9279 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:12.773314 9279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:12.773554 9279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:12.776171 9279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "5f42041bd2cc4800a6b4160f3a19cc2c"
format_stamp: "Formatted at 2026-06-19 03:01:12 on dist-test-slave-2gms"
server_key: "2a30bf2ac5bc17374fda35693b9d680d"
server_key_iv: "616c9a88ab53a1ea685b2ac293538495"
server_key_version: "encryptionkey@0"
I20260619 03:01:12.776880 9279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "5f42041bd2cc4800a6b4160f3a19cc2c"
format_stamp: "Formatted at 2026-06-19 03:01:12 on dist-test-slave-2gms"
server_key: "2a30bf2ac5bc17374fda35693b9d680d"
server_key_iv: "616c9a88ab53a1ea685b2ac293538495"
server_key_version: "encryptionkey@0"
I20260619 03:01:12.781517 9279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 03:01:12.784590 9294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:12.785763 9279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:12.785928 9279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "5f42041bd2cc4800a6b4160f3a19cc2c"
format_stamp: "Formatted at 2026-06-19 03:01:12 on dist-test-slave-2gms"
server_key: "2a30bf2ac5bc17374fda35693b9d680d"
server_key_iv: "616c9a88ab53a1ea685b2ac293538495"
server_key_version: "encryptionkey@0"
I20260619 03:01:12.786126 9279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:12.819339 9279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:12.822742 9279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:12.822981 9279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:12.823823 9279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:12.824977 9279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:12.825043 9279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:12.825232 9279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:12.825284 9279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:12.856683 9279 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:34017
I20260619 03:01:12.856760 9406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:34017 every 8 connection(s)
I20260619 03:01:12.857875 9279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:01:12.867481 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9279
I20260619 03:01:12.867668 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:01:12.868130 23448 external_mini_cluster.cc:1428] Setting key 001a9500ef963d1d65f01f4311b74227
Jun 19 03:01:12 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838072, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:12.877488 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:12.860280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:33351 (local address 127.22.230.62:36089)
0619 03:01:12.860682 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:12.860686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:12.861486 (+ 800us) server_negotiation.cc:407] Connection header received
0619 03:01:12.863169 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:12.863174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:12.863254 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:12.863362 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:12.864656 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:12.865356 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:12.866992 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:12.867573 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:12.867706 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:12.871109 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:12.871118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:12.871125 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:12.871165 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:12.874446 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:12.875136 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:12.875144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:12.875149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:12.875312 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:12.875685 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:12.875692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:12.875698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:12.876060 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:12.876217 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:12.876799 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 03:01:12.877033 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":222,"threads_started":1}
I20260619 03:01:12.877942 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:12.860727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36089 (local address 127.22.230.1:33351)
0619 03:01:12.861335 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:12.861375 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:12.862937 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:12.863569 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:12.863576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:12.864004 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:12.864497 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:12.864513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:12.865516 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:12.865520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:12.866814 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:12.866826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:12.867842 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:12.867849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:12.868043 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:12.868686 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:12.868709 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:12.870922 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:12.874621 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:12.874627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:12.874636 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:12.875012 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:12.875415 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:12.875417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:12.875420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:12.875576 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:12.876173 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:12.876177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:12.876451 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 03:01:12.876657 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 03:01:12.876965 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":241,"threads_started":1}
I20260619 03:01:12.879344 9407 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36089
I20260619 03:01:12.879619 9407 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:12.880331 9407 heartbeater.cc:511] Master 127.22.230.62:36089 requested a full tablet report, sending...
I20260619 03:01:12.882294 9218 ts_manager.cc:194] Registered new tserver with Master: 5f42041bd2cc4800a6b4160f3a19cc2c (127.22.230.1:34017)
I20260619 03:01:12.884002 9218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:33351
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 03:01:12.921605 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36089
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:13.224373 9413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:13.224611 9413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:13.224697 9413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:13.231567 9413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:13.231650 9413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:13.231871 9413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:01:13.240562 9413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:13.241708 9413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:13.242942 9413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:13.254973 9419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:13.254982 9418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:13.255007 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:13.255393 9413 server_base.cc:1034] running on GCE node
I20260619 03:01:13.256090 9413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:13.256875 9413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:13.258103 9413 hybrid_clock.cc:648] HybridClock initialized: now 1781838073258069 us; error 48 us; skew 500 ppm
I20260619 03:01:13.258379 9413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:13 dist-test-slave-2gms krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838073, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:13.261204 9413 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 03:01:13.262430 9413 webserver.cc:460] Webserver started at http://127.22.230.2:42377/ using document root <none> and password file <none>
I20260619 03:01:13.263077 9413 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:13.263206 9413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:13.263477 9413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:13.266098 9413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "345e420d821049eeb2206e20f6a04976"
format_stamp: "Formatted at 2026-06-19 03:01:13 on dist-test-slave-2gms"
server_key: "3771d14af833b7a8c611bf8e38b3a7e5"
server_key_iv: "6c020efe80e833f8105fd82779b5f5f1"
server_key_version: "encryptionkey@0"
I20260619 03:01:13.266878 9413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "345e420d821049eeb2206e20f6a04976"
format_stamp: "Formatted at 2026-06-19 03:01:13 on dist-test-slave-2gms"
server_key: "3771d14af833b7a8c611bf8e38b3a7e5"
server_key_iv: "6c020efe80e833f8105fd82779b5f5f1"
server_key_version: "encryptionkey@0"
I20260619 03:01:13.271397 9413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:01:13.274427 9428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:13.275624 9413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:13.275782 9413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "345e420d821049eeb2206e20f6a04976"
format_stamp: "Formatted at 2026-06-19 03:01:13 on dist-test-slave-2gms"
server_key: "3771d14af833b7a8c611bf8e38b3a7e5"
server_key_iv: "6c020efe80e833f8105fd82779b5f5f1"
server_key_version: "encryptionkey@0"
I20260619 03:01:13.275971 9413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:13.295706 9413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:13.299232 9413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:13.299517 9413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:13.300359 9413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:13.301560 9413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:13.301661 9413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:13.301795 9413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:13.301862 9413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:13.333009 9413 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:37459
I20260619 03:01:13.333083 9540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:37459 every 8 connection(s)
I20260619 03:01:13.334157 9413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:01:13.342087 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9413
I20260619 03:01:13.342257 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:01:13.342759 23448 external_mini_cluster.cc:1428] Setting key 1d5bfb60d2199d82ec3b95a412998dcf
Jun 19 03:01:13 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838073, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:13.353425 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.336479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:48811 (local address 127.22.230.62:36089)
0619 03:01:13.336670 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.336674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.337687 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 03:01:13.339455 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.339460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.339546 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.339652 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:13.341012 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.341747 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.343530 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.344041 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.344135 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.348047 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:13.348051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:13.348054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:13.348083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:13.350468 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:13.351361 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:13.351365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:13.351367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:13.351495 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:13.351949 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:13.351952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:13.351954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:13.352176 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:13.352284 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.353034 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.353185 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 03:01:13.354812 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.336946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36089 (local address 127.22.230.2:48811)
0619 03:01:13.337546 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.337579 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.339207 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.339877 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.339884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.340335 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:13.340838 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.340853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.341892 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.341896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.343376 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.343393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.344254 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.344262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.344556 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.345214 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:13.345234 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:13.347843 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:13.350613 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:13.350623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:13.350637 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:13.351247 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:13.351601 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:13.351607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:13.351621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:13.351844 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:13.352290 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:13.352300 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:13.352694 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.352959 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.353464 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260619 03:01:13.356096 9541 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36089
I20260619 03:01:13.356374 9541 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:13.357007 9541 heartbeater.cc:511] Master 127.22.230.62:36089 requested a full tablet report, sending...
I20260619 03:01:13.358335 9218 ts_manager.cc:194] Registered new tserver with Master: 345e420d821049eeb2206e20f6a04976 (127.22.230.2:37459)
I20260619 03:01:13.359125 9218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:48811
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 03:01:13.395681 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36089
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:13.697845 9546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:13.698077 9546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:13.698163 9546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:13.705103 9546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:13.705188 9546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:13.705392 9546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:01:13.714282 9546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:13.715382 9546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:13.716545 9546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:13.728456 9551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:13.728513 9552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:13.728523 9554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:13.729223 9546 server_base.cc:1034] running on GCE node
I20260619 03:01:13.729722 9546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:13.730508 9546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:13.731731 9546 hybrid_clock.cc:648] HybridClock initialized: now 1781838073731707 us; error 58 us; skew 500 ppm
I20260619 03:01:13.732000 9546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:13 dist-test-slave-2gms krb5kdc[9173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838073, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:13.738866 9546 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 03:01:13.740093 9546 webserver.cc:460] Webserver started at http://127.22.230.3:38219/ using document root <none> and password file <none>
I20260619 03:01:13.740636 9546 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:13.740729 9546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:13.740976 9546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:13.743507 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "21ccc93c382349a1941ff50ed0ad5b74"
format_stamp: "Formatted at 2026-06-19 03:01:13 on dist-test-slave-2gms"
server_key: "e02fa78197b01e0d2690fc767d838c7e"
server_key_iv: "3979261a4762d15edb08230997dc3d2d"
server_key_version: "encryptionkey@0"
I20260619 03:01:13.744127 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "21ccc93c382349a1941ff50ed0ad5b74"
format_stamp: "Formatted at 2026-06-19 03:01:13 on dist-test-slave-2gms"
server_key: "e02fa78197b01e0d2690fc767d838c7e"
server_key_iv: "3979261a4762d15edb08230997dc3d2d"
server_key_version: "encryptionkey@0"
I20260619 03:01:13.748656 9546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 03:01:13.751694 9561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:13.752912 9546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:01:13.753032 9546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "21ccc93c382349a1941ff50ed0ad5b74"
format_stamp: "Formatted at 2026-06-19 03:01:13 on dist-test-slave-2gms"
server_key: "e02fa78197b01e0d2690fc767d838c7e"
server_key_iv: "3979261a4762d15edb08230997dc3d2d"
server_key_version: "encryptionkey@0"
I20260619 03:01:13.753176 9546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:13.788014 9546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:13.792011 9546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:13.792212 9546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:13.792979 9546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:13.794075 9546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:13.794145 9546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:13.794220 9546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:13.794260 9546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:13.825398 9546 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:41427
I20260619 03:01:13.825439 9673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:41427 every 8 connection(s)
I20260619 03:01:13.826586 9546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:01:13.836288 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9546
I20260619 03:01:13.836465 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:01:13.836886 23448 external_mini_cluster.cc:1428] Setting key ca058dabbd9a34270cbad65c57a9a654
Jun 19 03:01:13 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838073, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:13.845189 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.829095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:55805 (local address 127.22.230.62:36089)
0619 03:01:13.829262 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.829266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.830343 (+ 1077us) server_negotiation.cc:407] Connection header received
0619 03:01:13.832098 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.832103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.832197 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.832295 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:13.833857 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.834653 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.836299 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.836832 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.836962 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.840454 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:13.840459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:13.840461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:13.840490 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:13.842778 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:13.843363 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:13.843366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:13.843368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:13.843497 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:13.843827 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:13.843829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:13.843831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:13.844071 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:13.844173 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.844798 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.844954 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:01:13.846004 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.829573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36089 (local address 127.22.230.3:55805)
0619 03:01:13.830194 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.830232 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.831842 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.832504 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.832511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.833001 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:13.833633 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.833656 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.834801 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.834805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.836156 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.836168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.837041 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.837050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.837297 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.837955 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:13.837976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:13.840247 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:13.842889 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:13.842894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:13.842905 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:13.843274 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:13.843580 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:13.843583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:13.843585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:13.843742 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:13.844164 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:13.844168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:13.844439 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.844706 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.845026 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":250,"threads_started":1}
I20260619 03:01:13.847275 9674 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36089
I20260619 03:01:13.847517 9674 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:13.848166 9674 heartbeater.cc:511] Master 127.22.230.62:36089 requested a full tablet report, sending...
I20260619 03:01:13.849356 9218 ts_manager.cc:194] Registered new tserver with Master: 21ccc93c382349a1941ff50ed0ad5b74 (127.22.230.3:41427)
I20260619 03:01:13.850236 9218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:55805
I20260619 03:01:13.851715 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:01:13.868348 9409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.855563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.22.230.62:36089)
0619 03:01:13.855734 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.855741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.856196 (+ 455us) server_negotiation.cc:407] Connection header received
0619 03:01:13.856383 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.856386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.856446 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.856531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:13.858282 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.858981 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.860764 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.861084 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.861167 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.863803 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:13.863807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:13.863809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:13.863834 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:13.865923 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:13.866682 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:13.866685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:13.866687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:13.866815 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:13.867174 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:13.867176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:13.867178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:13.867418 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:13.867600 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.867879 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.867998 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 03:01:13.871738 9217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:01:13.877898 9217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:01:13.887368 9407 heartbeater.cc:503] Master 127.22.230.62:36089 was elected leader, sending a full tablet report...
I20260619 03:01:13.900063 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.890796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:34017 (local address 127.0.0.1:37718)
0619 03:01:13.891418 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.891437 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.891580 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.892732 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.892736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.892762 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:13.893154 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.893166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.894926 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.894933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.896932 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.896944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.898726 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.898732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.899090 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.899160 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.899270 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.899478 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":163,"thread_start_us":456,"threads_started":1}
I20260619 03:01:13.900522 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.891444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.22.230.3:41427)
0619 03:01:13.892486 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.892494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.893143 (+ 649us) server_negotiation.cc:407] Connection header received
0619 03:01:13.893235 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.893240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.893372 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.893531 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:13.894087 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.895662 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.897804 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.898725 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.898845 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.899081 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.899991 (+ 910us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.900149 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":867,"threads_started":1}
I20260619 03:01:13.900696 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.891136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.22.230.1:34017)
0619 03:01:13.892203 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.892211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.892239 (+ 28us) server_negotiation.cc:407] Connection header received
0619 03:01:13.892313 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.892320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.892538 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.893325 (+ 787us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:13.893503 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.894808 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.897134 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.898459 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.899946 (+ 1487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.900134 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.900222 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.900368 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":771,"threads_started":1}
I20260619 03:01:13.900799 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.890944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:41427 (local address 127.0.0.1:36120)
0619 03:01:13.892889 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.892907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.893017 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.893524 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.893528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.893550 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:13.893926 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.893935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.895835 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.895840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.897666 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.897677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.898959 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.898964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.899171 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.899196 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.899271 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.899478 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1805,"mutex_wait_us":115,"thread_start_us":2074,"threads_started":1}
I20260619 03:01:13.901494 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.889126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:37459 (local address 127.0.0.1:37388)
0619 03:01:13.889901 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.889942 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.890135 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.892026 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.892035 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.892070 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:13.892561 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.892576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.894310 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.894324 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.896897 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.896920 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.899834 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.899844 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.900840 (+ 996us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.900873 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.901000 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.901144 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":122,"thread_start_us":270,"threads_started":1}
I20260619 03:01:13.901537 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.889335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.22.230.2:37459)
0619 03:01:13.891143 (+ 1808us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.891153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.891181 (+ 28us) server_negotiation.cc:407] Connection header received
0619 03:01:13.891288 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.891297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.891552 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.891751 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:13.892735 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.894180 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.897079 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.898453 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.898610 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.898812 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.901049 (+ 2237us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.901222 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1658,"thread_start_us":1573,"threads_started":1}
I20260619 03:01:13.904109 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 3cce131a9154421f98f949d74f9032ca (DEFAULT_TABLE table=test-table [id=33ce1ba5648f4706b0bd40ce85382036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:13.904349 9609 tablet_service.cc:1467] Processing CreateTablet for tablet 3cce131a9154421f98f949d74f9032ca (DEFAULT_TABLE table=test-table [id=33ce1ba5648f4706b0bd40ce85382036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:13.904838 9476 tablet_service.cc:1467] Processing CreateTablet for tablet 3cce131a9154421f98f949d74f9032ca (DEFAULT_TABLE table=test-table [id=33ce1ba5648f4706b0bd40ce85382036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:13.905340 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cce131a9154421f98f949d74f9032ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:13.905762 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cce131a9154421f98f949d74f9032ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:13.906095 9476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cce131a9154421f98f949d74f9032ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:13.915951 9692 tablet_bootstrap.cc:492] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Bootstrap starting.
I20260619 03:01:13.919101 9692 tablet_bootstrap.cc:654] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:13.919242 9693 tablet_bootstrap.cc:492] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Bootstrap starting.
I20260619 03:01:13.919983 9694 tablet_bootstrap.cc:492] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Bootstrap starting.
I20260619 03:01:13.921574 9692 log.cc:824] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:13.923355 9693 tablet_bootstrap.cc:654] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:13.923399 9694 tablet_bootstrap.cc:654] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:13.924279 9693 log.cc:824] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:13.924548 9692 tablet_bootstrap.cc:492] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: No bootstrap required, opened a new log
I20260619 03:01:13.924512 9694 log.cc:824] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:13.924772 9692 ts_tablet_manager.cc:1397] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 03:01:13.927441 9693 tablet_bootstrap.cc:492] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: No bootstrap required, opened a new log
I20260619 03:01:13.927441 9694 tablet_bootstrap.cc:492] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: No bootstrap required, opened a new log
I20260619 03:01:13.927686 9694 ts_tablet_manager.cc:1397] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 03:01:13.927686 9693 ts_tablet_manager.cc:1397] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 03:01:13.932301 9692 raft_consensus.cc:348] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.932616 9692 raft_consensus.cc:374] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:13.932695 9692 raft_consensus.cc:729] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f42041bd2cc4800a6b4160f3a19cc2c, State: Initialized, Role: FOLLOWER
I20260619 03:01:13.933495 9692 consensus_queue.cc:260] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.933849 9698 raft_consensus.cc:482] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:13.934129 9698 raft_consensus.cc:504] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.937292 9693 raft_consensus.cc:348] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.937294 9694 raft_consensus.cc:348] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.937671 9694 raft_consensus.cc:374] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:13.937674 9693 raft_consensus.cc:374] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:13.937763 9693 raft_consensus.cc:729] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345e420d821049eeb2206e20f6a04976, State: Initialized, Role: FOLLOWER
I20260619 03:01:13.937767 9694 raft_consensus.cc:729] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ccc93c382349a1941ff50ed0ad5b74, State: Initialized, Role: FOLLOWER
I20260619 03:01:13.938297 9694 consensus_queue.cc:260] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.938297 9693 consensus_queue.cc:260] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.943190 9692 ts_tablet_manager.cc:1428] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260619 03:01:13.944113 9698 leader_election.cc:288] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345e420d821049eeb2206e20f6a04976 (127.22.230.2:37459), 21ccc93c382349a1941ff50ed0ad5b74 (127.22.230.3:41427)
I20260619 03:01:13.945721 9541 heartbeater.cc:503] Master 127.22.230.62:36089 was elected leader, sending a full tablet report...
I20260619 03:01:13.946616 9693 ts_tablet_manager.cc:1428] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260619 03:01:13.947633 9694 ts_tablet_manager.cc:1428] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260619 03:01:13.947947 9674 heartbeater.cc:503] Master 127.22.230.62:36089 was elected leader, sending a full tablet report...
I20260619 03:01:13.955094 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.936557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:37459 (local address 127.22.230.1:41097)
0619 03:01:13.940735 (+ 4178us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.940754 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.941282 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.943896 (+ 2614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.943903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.943944 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:13.944300 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.944317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.949372 (+ 5055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.949380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.952175 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.952196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.954320 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.954327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.954505 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.954559 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.954677 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.954815 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4031,"thread_start_us":3630,"threads_started":1}
I20260619 03:01:13.955149 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.936549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:41097 (local address 127.22.230.2:37459)
0619 03:01:13.936753 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.936761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.940820 (+ 4059us) server_negotiation.cc:407] Connection header received
0619 03:01:13.943481 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.943490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.943586 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.943664 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:13.947819 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.949148 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.953033 (+ 3885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.954144 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.954252 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.954297 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.954737 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.954889 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 03:01:13.956061 9496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cce131a9154421f98f949d74f9032ca" candidate_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345e420d821049eeb2206e20f6a04976" is_pre_election: true
I20260619 03:01:13.956527 9496 raft_consensus.cc:2454] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f42041bd2cc4800a6b4160f3a19cc2c in term 0.
I20260619 03:01:13.957090 9298 leader_election.cc:302] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345e420d821049eeb2206e20f6a04976, 5f42041bd2cc4800a6b4160f3a19cc2c; no voters:
I20260619 03:01:13.957463 9699 raft_consensus.cc:2781] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:01:13.957608 9699 raft_consensus.cc:482] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:01:13.957747 9699 raft_consensus.cc:3037] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:13.958791 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.940902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:41427 (local address 127.22.230.1:47821)
0619 03:01:13.952714 (+ 11812us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:13.952731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:13.952849 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:13.953263 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:13.953268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:13.953294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:13.953642 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.953652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.954832 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.954838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.956584 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:13.956608 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.958176 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.958180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:13.958288 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.958309 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 03:01:13.958378 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 03:01:13.958477 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11672,"thread_start_us":11638,"threads_started":1}
I20260619 03:01:13.958806 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.941005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:47821 (local address 127.22.230.3:41427)
0619 03:01:13.941233 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.941242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.952852 (+ 11610us) server_negotiation.cc:407] Connection header received
0619 03:01:13.952951 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.952955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.953040 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.953126 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:13.953780 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.954708 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.956751 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.957990 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.958155 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.958240 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.958433 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.958559 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:01:13.959683 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cce131a9154421f98f949d74f9032ca" candidate_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ccc93c382349a1941ff50ed0ad5b74" is_pre_election: true
I20260619 03:01:13.960073 9629 raft_consensus.cc:2454] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f42041bd2cc4800a6b4160f3a19cc2c in term 0.
I20260619 03:01:13.960640 9699 raft_consensus.cc:504] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.961212 9699 leader_election.cc:288] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [CANDIDATE]: Term 1 election: Requested vote from peers 345e420d821049eeb2206e20f6a04976 (127.22.230.2:37459), 21ccc93c382349a1941ff50ed0ad5b74 (127.22.230.3:41427)
I20260619 03:01:13.961634 9496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cce131a9154421f98f949d74f9032ca" candidate_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345e420d821049eeb2206e20f6a04976"
I20260619 03:01:13.961714 9629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cce131a9154421f98f949d74f9032ca" candidate_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ccc93c382349a1941ff50ed0ad5b74"
I20260619 03:01:13.961800 9496 raft_consensus.cc:3037] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:13.961927 9629 raft_consensus.cc:3037] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:13.963799 9496 raft_consensus.cc:2454] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f42041bd2cc4800a6b4160f3a19cc2c in term 1.
I20260619 03:01:13.963949 9629 raft_consensus.cc:2454] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f42041bd2cc4800a6b4160f3a19cc2c in term 1.
I20260619 03:01:13.964154 9298 leader_election.cc:302] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345e420d821049eeb2206e20f6a04976, 5f42041bd2cc4800a6b4160f3a19cc2c; no voters:
I20260619 03:01:13.964383 9699 raft_consensus.cc:2781] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:13.964553 9699 raft_consensus.cc:686] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 1 LEADER]: Becoming Leader. State: Replica: 5f42041bd2cc4800a6b4160f3a19cc2c, State: Running, Role: LEADER
I20260619 03:01:13.964936 9699 consensus_queue.cc:237] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } }
I20260619 03:01:13.969197 9217 catalog_manager.cc:5495] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f42041bd2cc4800a6b4160f3a19cc2c (127.22.230.1). New cstate: current_term: 1 leader_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f42041bd2cc4800a6b4160f3a19cc2c" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 34017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 } health_report { overall_health: UNKNOWN } } }
I20260619 03:01:13.997838 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:13.993101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.22.230.1:34017)
0619 03:01:13.993256 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:13.993260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:13.993356 (+ 96us) server_negotiation.cc:407] Connection header received
0619 03:01:13.993496 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:13.993499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:13.993566 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:13.993668 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:01:13.993999 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.994699 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.996066 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:13.996440 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:13.996524 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:13.996954 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:01:13.997402 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:01:13.997517 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:13.997617 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 03:01:13.997651 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 03:01:14.004874 9629 raft_consensus.cc:1261] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 1 FOLLOWER]: Refusing update from remote peer 5f42041bd2cc4800a6b4160f3a19cc2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:14.004869 9496 raft_consensus.cc:1261] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 1 FOLLOWER]: Refusing update from remote peer 5f42041bd2cc4800a6b4160f3a19cc2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:14.005806 9698 consensus_queue.cc:1035] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ccc93c382349a1941ff50ed0ad5b74" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:14.006137 9699 consensus_queue.cc:1035] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "345e420d821049eeb2206e20f6a04976" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 37459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:14.016691 9706 mvcc.cc:205] Tried to move back new op lower bound from 7298408751115644928 to 7298408750964142080. Current Snapshot: MvccSnapshot[applied={T|T < 7298408751115644928}]
I20260619 03:01:14.021603 9708 mvcc.cc:205] Tried to move back new op lower bound from 7298408751115644928 to 7298408750964142080. Current Snapshot: MvccSnapshot[applied={T|T < 7298408751115644928}]
I20260619 03:01:14.030256 9218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:14.030493 9218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:14.033725 9218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68cc0592439b492faccf443852c398c7: Sending DeleteTablet for 3 replicas of tablet 3cce131a9154421f98f949d74f9032ca
I20260619 03:01:14.034788 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cce131a9154421f98f949d74f9032ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:14 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:37718
I20260619 03:01:14.034848 9609 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cce131a9154421f98f949d74f9032ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:14 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:36120
I20260619 03:01:14.035620 9716 tablet_replica.cc:330] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: stopping tablet replica
I20260619 03:01:14.036041 9716 raft_consensus.cc:2229] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:14.036558 9716 raft_consensus.cc:2258] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:14.038705 9716 ts_tablet_manager.cc:1905] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:14.039651 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9279
I20260619 03:01:14.043229 9476 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cce131a9154421f98f949d74f9032ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:14 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:37388
I20260619 03:01:14.043916 9717 tablet_replica.cc:330] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: stopping tablet replica
I20260619 03:01:14.044401 9717 raft_consensus.cc:2229] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:01:14.045083 9717 raft_consensus.cc:2258] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:14.048496 9716 ts_tablet_manager.cc:1918] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:14.048684 9716 log.cc:1196] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/3cce131a9154421f98f949d74f9032ca
I20260619 03:01:14.049232 9716 ts_tablet_manager.cc:1939] T 3cce131a9154421f98f949d74f9032ca P 21ccc93c382349a1941ff50ed0ad5b74: Deleting consensus metadata
I20260619 03:01:14.052332 9719 tablet_replica.cc:330] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: stopping tablet replica
I20260619 03:01:14.052580 9717 ts_tablet_manager.cc:1905] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:14.052721 9719 raft_consensus.cc:2229] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:14.053181 9719 raft_consensus.cc:2258] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:14.053972 9205 catalog_manager.cc:4841] TS 21ccc93c382349a1941ff50ed0ad5b74 (127.22.230.3:41427): tablet 3cce131a9154421f98f949d74f9032ca (table test-table [id=33ce1ba5648f4706b0bd40ce85382036]) successfully deleted
I20260619 03:01:14.056159 9719 ts_tablet_manager.cc:1905] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:14.060113 9717 ts_tablet_manager.cc:1918] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:14.060266 9717 log.cc:1196] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/3cce131a9154421f98f949d74f9032ca
Jun 19 03:01:14 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 03:01:14.060829 9717 ts_tablet_manager.cc:1939] T 3cce131a9154421f98f949d74f9032ca P 5f42041bd2cc4800a6b4160f3a19cc2c: Deleting consensus metadata
I20260619 03:01:14.062532 9202 catalog_manager.cc:4841] TS 5f42041bd2cc4800a6b4160f3a19cc2c (127.22.230.1:34017): tablet 3cce131a9154421f98f949d74f9032ca (table test-table [id=33ce1ba5648f4706b0bd40ce85382036]) successfully deleted
I20260619 03:01:14.067108 9719 ts_tablet_manager.cc:1918] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:14.067260 9719 log.cc:1196] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/3cce131a9154421f98f949d74f9032ca
I20260619 03:01:14.067788 9719 ts_tablet_manager.cc:1939] T 3cce131a9154421f98f949d74f9032ca P 345e420d821049eeb2206e20f6a04976: Deleting consensus metadata
I20260619 03:01:14.069389 9205 catalog_manager.cc:4841] TS 345e420d821049eeb2206e20f6a04976 (127.22.230.2:37459): tablet 3cce131a9154421f98f949d74f9032ca (table test-table [id=33ce1ba5648f4706b0bd40ce85382036]) successfully deleted
I20260619 03:01:14.071151 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:14.041105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.22.230.1:34017)
0619 03:01:14.041311 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:14.041320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:14.049179 (+ 7859us) server_negotiation.cc:407] Connection header received
0619 03:01:14.049433 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:14.049436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:14.049505 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:14.049609 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:14.050953 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:14.052432 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:14.054829 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:14.055513 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:14.055602 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:14.062177 (+ 6575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:14.062191 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:14.062206 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:14.062646 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:14.068162 (+ 5516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:14.069001 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:14.069011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:14.069017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:14.069219 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:14.069668 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:14.069673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:14.069676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:14.069980 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:14.070291 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:14.070551 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 03:01:14.070799 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 03:01:14.071858 9402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:14.165783 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9279
I20260619 03:01:14.183564 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9413
Jun 19 03:01:14 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 03:01:14.197835 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:14.184288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.22.230.2:37459)
0619 03:01:14.184481 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:14.184486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:14.184506 (+ 20us) server_negotiation.cc:407] Connection header received
0619 03:01:14.184674 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:14.184678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:14.184738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:14.184839 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:14.186070 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:14.186881 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:14.188130 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:14.188518 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:14.188654 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:14.192791 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:14.192803 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:14.192817 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:14.192853 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:14.195260 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:14.195917 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:14.195930 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:14.195932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:14.196050 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:14.196399 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:14.196415 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:14.196422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:14.196768 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:14.197040 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:14.197338 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 03:01:14.197543 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 03:01:14.198524 9536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:14.290005 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9413
I20260619 03:01:14.306607 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9546
Jun 19 03:01:14 dist-test-slave-2gms krb5kdc[9173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 03:01:14.320996 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:14.307360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.22.230.3:41427)
0619 03:01:14.307546 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:14.307553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:14.307573 (+ 20us) server_negotiation.cc:407] Connection header received
0619 03:01:14.307754 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:14.307760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:14.307836 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:14.307954 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:14.309274 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:14.310012 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:14.311443 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:14.311825 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:14.311960 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:14.316151 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:14.316164 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:14.316179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:14.316209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:14.318774 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:14.319367 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:14.319373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:14.319376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:14.319498 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:14.319786 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:14.319789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:14.319791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:14.320089 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:14.320282 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:14.320512 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 03:01:14.320662 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:01:14.321697 9669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:14.406960 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9546
I20260619 03:01:14.423645 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9186
I20260619 03:01:14.424616 9248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:14.530189 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9186
2026-06-19T03:01:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3944 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T03:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:14Z Disabled control of system clock
I20260619 03:01:14.604058 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:42089
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 03:01:14.909696 9727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:14.909943 9727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:14.910038 9727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:14.916785 9727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:14.916870 9727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:14.916904 9727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:14.916968 9727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:14.917003 9727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:14.925192 9727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:42089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:14.926299 9727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:14.927945 9727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:14.938894 9735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:14.938951 9733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:14.938948 9732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:14.939484 9727 server_base.cc:1034] running on GCE node
I20260619 03:01:14.940208 9727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:14.941466 9727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:14.942677 9727 hybrid_clock.cc:648] HybridClock initialized: now 1781838074942648 us; error 37 us; skew 500 ppm
I20260619 03:01:14.942999 9727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:14.945001 9727 webserver.cc:460] Webserver started at http://127.22.230.62:39431/ using document root <none> and password file <none>
I20260619 03:01:14.945642 9727 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:14.945771 9727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:14.946089 9727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:14.948853 9727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "e0c03a1798634e68baf15e552a29d671"
format_stamp: "Formatted at 2026-06-19 03:01:14 on dist-test-slave-2gms"
server_key: "a217cdc05bc855f59267bd7f893a0352"
server_key_iv: "621d27804b8ad15eb0023deca05148ae"
server_key_version: "encryptionkey@0"
I20260619 03:01:14.949604 9727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "e0c03a1798634e68baf15e552a29d671"
format_stamp: "Formatted at 2026-06-19 03:01:14 on dist-test-slave-2gms"
server_key: "a217cdc05bc855f59267bd7f893a0352"
server_key_iv: "621d27804b8ad15eb0023deca05148ae"
server_key_version: "encryptionkey@0"
I20260619 03:01:14.954273 9727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260619 03:01:14.957445 9741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:14.958706 9727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 03:01:14.958892 9727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "e0c03a1798634e68baf15e552a29d671"
format_stamp: "Formatted at 2026-06-19 03:01:14 on dist-test-slave-2gms"
server_key: "a217cdc05bc855f59267bd7f893a0352"
server_key_iv: "621d27804b8ad15eb0023deca05148ae"
server_key_version: "encryptionkey@0"
I20260619 03:01:14.959098 9727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:14.983661 9727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:14.984441 9727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:14.984719 9727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:15.002580 9727 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:42089
I20260619 03:01:15.002630 9792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:42089 every 8 connection(s)
I20260619 03:01:15.003878 9727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:15.007319 9793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:15.013520 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9727
I20260619 03:01:15.013687 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:15.014073 23448 external_mini_cluster.cc:1428] Setting key 883de7ea71e27fdfb84d9755a3102978
I20260619 03:01:15.016873 9793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671: Bootstrap starting.
I20260619 03:01:15.020023 9793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:15.021049 9793 log.cc:824] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:15.023784 9793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671: No bootstrap required, opened a new log
I20260619 03:01:15.027240 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:15.015904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.22.230.62:42089)
0619 03:01:15.016851 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:15.016864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:15.016902 (+ 38us) server_negotiation.cc:407] Connection header received
0619 03:01:15.018624 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:15.018645 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:15.019040 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:15.019316 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:15.019906 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.021301 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.023055 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.023698 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.023862 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:15.025013 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:15.025019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:15.025031 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:15.025537 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:15.025614 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:15.025641 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:15.025726 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:15.025978 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 03:01:15.026291 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":335,"threads_started":1}
I20260619 03:01:15.031632 9793 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c03a1798634e68baf15e552a29d671" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42089 } }
I20260619 03:01:15.031944 9793 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:15.032055 9793 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c03a1798634e68baf15e552a29d671, State: Initialized, Role: FOLLOWER
I20260619 03:01:15.032600 9793 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c03a1798634e68baf15e552a29d671" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42089 } }
I20260619 03:01:15.032819 9793 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:15.032943 9793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:15.033111 9793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:15.035121 9793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c03a1798634e68baf15e552a29d671" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42089 } }
I20260619 03:01:15.035576 9793 leader_election.cc:302] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c03a1798634e68baf15e552a29d671; no voters:
I20260619 03:01:15.036110 9793 leader_election.cc:288] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:15.036255 9798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:15.036968 9798 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [term 1 LEADER]: Becoming Leader. State: Replica: e0c03a1798634e68baf15e552a29d671, State: Running, Role: LEADER
I20260619 03:01:15.037505 9798 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c03a1798634e68baf15e552a29d671" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42089 } }
I20260619 03:01:15.038040 9793 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:15.040359 9800 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c03a1798634e68baf15e552a29d671. Latest consensus state: current_term: 1 leader_uuid: "e0c03a1798634e68baf15e552a29d671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c03a1798634e68baf15e552a29d671" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42089 } } }
I20260619 03:01:15.040799 9800 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:15.040359 9799 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c03a1798634e68baf15e552a29d671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c03a1798634e68baf15e552a29d671" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 42089 } } }
I20260619 03:01:15.041102 9799 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:15.047613 9804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:15.052708 9804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:15.060276 9804 catalog_manager.cc:1347] Generated new cluster ID: 26eb3dcf399649ccb55e45d83ef4f89f
I20260619 03:01:15.060369 9804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:15.103750 9804 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:15.105536 9804 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:15.115450 9804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671: Generated new TSK 0
I20260619 03:01:15.116202 9804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:15.128968 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42089
--builtin_ntp_servers=127.22.230.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 03:01:15.454332 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:15.454618 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:15.454746 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:15.463033 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:15.463109 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:15.463202 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:01:15.471642 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:15.472656 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:15.474117 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:15.485644 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:15.485611 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:15.485906 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:15.486865 9817 server_base.cc:1034] running on GCE node
I20260619 03:01:15.487357 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:15.488277 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:15.489475 9817 hybrid_clock.cc:648] HybridClock initialized: now 1781838075489433 us; error 31 us; skew 500 ppm
I20260619 03:01:15.489768 9817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:15.491780 9817 webserver.cc:460] Webserver started at http://127.22.230.1:43229/ using document root <none> and password file <none>
I20260619 03:01:15.492414 9817 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:15.492529 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:15.492810 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:15.495592 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "4d4e8a2ef4bd46b09f81e4eda941e835"
format_stamp: "Formatted at 2026-06-19 03:01:15 on dist-test-slave-2gms"
server_key: "ea4004e13e30bacb5aa665f810eebd9a"
server_key_iv: "a18f5ee8e3807015b5b98c93282364bd"
server_key_version: "encryptionkey@0"
I20260619 03:01:15.496392 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "4d4e8a2ef4bd46b09f81e4eda941e835"
format_stamp: "Formatted at 2026-06-19 03:01:15 on dist-test-slave-2gms"
server_key: "ea4004e13e30bacb5aa665f810eebd9a"
server_key_iv: "a18f5ee8e3807015b5b98c93282364bd"
server_key_version: "encryptionkey@0"
I20260619 03:01:15.501040 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:01:15.504089 9831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:15.505249 9817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:01:15.505420 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "4d4e8a2ef4bd46b09f81e4eda941e835"
format_stamp: "Formatted at 2026-06-19 03:01:15 on dist-test-slave-2gms"
server_key: "ea4004e13e30bacb5aa665f810eebd9a"
server_key_iv: "a18f5ee8e3807015b5b98c93282364bd"
server_key_version: "encryptionkey@0"
I20260619 03:01:15.505648 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:15.524089 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:15.524717 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:15.525038 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:15.525867 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:15.527200 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:15.527323 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:15.527467 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:15.527554 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:15.558538 9817 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:41207
I20260619 03:01:15.558692 9943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:41207 every 8 connection(s)
I20260619 03:01:15.559748 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:01:15.566907 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9817
I20260619 03:01:15.567102 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:01:15.567523 23448 external_mini_cluster.cc:1428] Setting key c06a2ecb141a90e1708c4fd23ac497b0
I20260619 03:01:15.570176 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42089
--builtin_ntp_servers=127.22.230.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 03:01:15.572418 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:15.562235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:47775 (local address 127.22.230.62:42089)
0619 03:01:15.562678 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:15.562682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:15.563823 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 03:01:15.565533 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:15.565541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:15.565634 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:15.565755 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:15.566794 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.567681 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.569697 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.570115 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.570239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:15.571622 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:15.571628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:15.571633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:15.571651 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:15.571696 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:15.571707 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:15.571796 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:15.572025 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 03:01:15.572105 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":205,"threads_started":1}
I20260619 03:01:15.573194 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:15.562868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42089 (local address 127.22.230.1:47775)
0619 03:01:15.563684 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:15.563716 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:15.565264 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:15.565972 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:15.565979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:15.566060 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:15.566586 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:15.566632 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.567836 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.567845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:15.569567 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:15.569579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.570310 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.570316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:15.570661 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:15.571326 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:15.571328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:15.571361 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:15.571440 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:15.571794 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:15.571805 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:15.571823 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 03:01:15.571983 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 03:01:15.572324 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":257,"threads_started":1}
I20260619 03:01:15.574488 9944 heartbeater.cc:347] Connected to a master server at 127.22.230.62:42089
I20260619 03:01:15.574836 9944 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:15.575723 9944 heartbeater.cc:511] Master 127.22.230.62:42089 requested a full tablet report, sending...
I20260619 03:01:15.577687 9758 ts_manager.cc:194] Registered new tserver with Master: 4d4e8a2ef4bd46b09f81e4eda941e835 (127.22.230.1:41207)
I20260619 03:01:15.579165 9758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.1:47775
W20260619 03:01:15.873793 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:15.874027 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:15.874104 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:15.880980 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:15.881078 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:15.881235 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:01:15.889549 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:15.890756 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:15.892418 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:15.904461 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:15.904493 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:15.904542 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:15.905403 9948 server_base.cc:1034] running on GCE node
I20260619 03:01:15.905920 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:15.906762 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:15.908015 9948 hybrid_clock.cc:648] HybridClock initialized: now 1781838075907940 us; error 39 us; skew 500 ppm
I20260619 03:01:15.908318 9948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:15.910351 9948 webserver.cc:460] Webserver started at http://127.22.230.2:43263/ using document root <none> and password file <none>
I20260619 03:01:15.911019 9948 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:15.911147 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:15.911434 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:15.914019 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "cc71048acf8a47d6b753541fa8f1cab2"
format_stamp: "Formatted at 2026-06-19 03:01:15 on dist-test-slave-2gms"
server_key: "b97a22fc006a1c6dd08bfbde24f29a45"
server_key_iv: "054675bc786817c4dfbea598ef00cd0c"
server_key_version: "encryptionkey@0"
I20260619 03:01:15.914805 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "cc71048acf8a47d6b753541fa8f1cab2"
format_stamp: "Formatted at 2026-06-19 03:01:15 on dist-test-slave-2gms"
server_key: "b97a22fc006a1c6dd08bfbde24f29a45"
server_key_iv: "054675bc786817c4dfbea598ef00cd0c"
server_key_version: "encryptionkey@0"
I20260619 03:01:15.919302 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:01:15.922398 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:15.923616 9948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:15.923781 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "cc71048acf8a47d6b753541fa8f1cab2"
format_stamp: "Formatted at 2026-06-19 03:01:15 on dist-test-slave-2gms"
server_key: "b97a22fc006a1c6dd08bfbde24f29a45"
server_key_iv: "054675bc786817c4dfbea598ef00cd0c"
server_key_version: "encryptionkey@0"
I20260619 03:01:15.924002 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:15.946419 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:15.947275 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:15.947548 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:15.948460 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:15.949738 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:15.949860 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:15.949980 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:15.950064 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:15.979868 9948 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:33231
I20260619 03:01:15.979948 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:33231 every 8 connection(s)
I20260619 03:01:15.981017 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:01:15.986258 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 9948
I20260619 03:01:15.986433 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:01:15.986909 23448 external_mini_cluster.cc:1428] Setting key 935008d62a403647faa1d1f40ed8b06f
I20260619 03:01:15.989475 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42089
--builtin_ntp_servers=127.22.230.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 03:01:15.993798 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:15.983388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:56741 (local address 127.22.230.62:42089)
0619 03:01:15.983571 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:15.983576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:15.984823 (+ 1247us) server_negotiation.cc:407] Connection header received
0619 03:01:15.986622 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:15.986627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:15.986725 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:15.986849 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:15.987931 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.988787 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.990793 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.991539 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.991676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:15.992931 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:15.992940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:15.992947 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:15.992970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:15.993011 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:15.993030 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:15.993126 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:15.993305 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 03:01:15.993420 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 03:01:15.994529 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:15.983891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42089 (local address 127.22.230.2:56741)
0619 03:01:15.984682 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:15.984713 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:15.986286 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:15.987073 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:15.987081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:15.987168 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:15.987759 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:15.987783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.988940 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.988951 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:15.990641 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:15.990655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:15.991717 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:15.991721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:15.991899 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:15.992643 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:15.992645 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:15.992680 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:15.992750 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:15.993126 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:15.993133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:15.993138 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 03:01:15.993265 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 03:01:15.993525 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":244,"threads_started":1}
I20260619 03:01:15.995977 10075 heartbeater.cc:347] Connected to a master server at 127.22.230.62:42089
I20260619 03:01:15.996281 10075 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:15.997130 10075 heartbeater.cc:511] Master 127.22.230.62:42089 requested a full tablet report, sending...
I20260619 03:01:15.998620 9758 ts_manager.cc:194] Registered new tserver with Master: cc71048acf8a47d6b753541fa8f1cab2 (127.22.230.2:33231)
I20260619 03:01:15.999341 9758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.2:56741
W20260619 03:01:16.298902 10078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:16.299130 10078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:16.299266 10078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:16.305424 10078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:16.305493 10078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:16.305657 10078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:01:16.314049 10078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:16.315179 10078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:16.316773 10078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:16.328933 10084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:16.328995 10083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:16.328995 10086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:16.330024 10078 server_base.cc:1034] running on GCE node
I20260619 03:01:16.330549 10078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:16.331439 10078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:16.332633 10078 hybrid_clock.cc:648] HybridClock initialized: now 1781838076332590 us; error 36 us; skew 500 ppm
I20260619 03:01:16.332906 10078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:16.334860 10078 webserver.cc:460] Webserver started at http://127.22.230.3:36255/ using document root <none> and password file <none>
I20260619 03:01:16.335487 10078 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:16.335629 10078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:16.335925 10078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:16.338634 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "dad8f9b8c7f34f9cb16570b4230c451c"
format_stamp: "Formatted at 2026-06-19 03:01:16 on dist-test-slave-2gms"
server_key: "a932c2ed977e250261678a7bda720479"
server_key_iv: "1e2e25510010ed635fe523ac8896acc2"
server_key_version: "encryptionkey@0"
I20260619 03:01:16.339435 10078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "dad8f9b8c7f34f9cb16570b4230c451c"
format_stamp: "Formatted at 2026-06-19 03:01:16 on dist-test-slave-2gms"
server_key: "a932c2ed977e250261678a7bda720479"
server_key_iv: "1e2e25510010ed635fe523ac8896acc2"
server_key_version: "encryptionkey@0"
I20260619 03:01:16.343920 10078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 03:01:16.347016 10092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:16.348235 10078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 03:01:16.348408 10078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "dad8f9b8c7f34f9cb16570b4230c451c"
format_stamp: "Formatted at 2026-06-19 03:01:16 on dist-test-slave-2gms"
server_key: "a932c2ed977e250261678a7bda720479"
server_key_iv: "1e2e25510010ed635fe523ac8896acc2"
server_key_version: "encryptionkey@0"
I20260619 03:01:16.348613 10078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:16.364876 10078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:16.365552 10078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:16.365852 10078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:16.366779 10078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:16.368021 10078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:16.368139 10078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:16.368263 10078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:16.368350 10078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:16.398900 10078 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:39859
I20260619 03:01:16.398968 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:39859 every 8 connection(s)
I20260619 03:01:16.400003 10078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:01:16.406279 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10078
I20260619 03:01:16.406464 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:01:16.406934 23448 external_mini_cluster.cc:1428] Setting key 8318e8c7bd540f284b4da051f0582e53
I20260619 03:01:16.411523 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.402199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:47535 (local address 127.22.230.62:42089)
0619 03:01:16.402369 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.402373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.403586 (+ 1213us) server_negotiation.cc:407] Connection header received
0619 03:01:16.405253 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.405260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.405353 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.405462 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:16.406505 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.407348 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.408937 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.409352 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.409497 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.410762 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:16.410766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:16.410770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:16.410785 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:16.410813 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:16.410826 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:16.410917 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.411135 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.411218 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 03:01:16.412282 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.402681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:42089 (local address 127.22.230.3:47535)
0619 03:01:16.403448 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:16.403479 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:16.405000 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:16.405699 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:16.405707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:16.405785 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:16.406306 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.406323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.407499 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.407505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.408759 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.408790 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.409572 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.409579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.409796 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.410434 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:16.410437 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:16.410470 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:16.410539 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:16.410927 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:16.410939 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:16.410946 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 03:01:16.411125 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 03:01:16.411435 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":233,"threads_started":1}
I20260619 03:01:16.413389 10205 heartbeater.cc:347] Connected to a master server at 127.22.230.62:42089
I20260619 03:01:16.413646 10205 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:16.414260 10205 heartbeater.cc:511] Master 127.22.230.62:42089 requested a full tablet report, sending...
I20260619 03:01:16.415467 9758 ts_manager.cc:194] Registered new tserver with Master: dad8f9b8c7f34f9cb16570b4230c451c (127.22.230.3:39859)
I20260619 03:01:16.416214 9758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.3:47535
I20260619 03:01:16.421167 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:01:16.430660 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.424772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.22.230.62:42089)
0619 03:01:16.424979 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.424987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.425288 (+ 301us) server_negotiation.cc:407] Connection header received
0619 03:01:16.425467 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.425473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.425546 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.425670 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:16.426149 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.426939 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.428347 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.428720 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.428856 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.429937 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:16.429942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:16.429945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:16.429960 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:16.429986 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:16.429997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:16.430137 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.430315 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.430368 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 03:01:16.433970 9758 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:01:16.440773 9758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:01:16.463749 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.451467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60640)
0619 03:01:16.452128 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:16.452164 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:16.452349 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:16.453466 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:16.453473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:16.453503 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:16.453949 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.453962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.456417 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.456426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.459582 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.459603 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.462458 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.462466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.462770 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.462855 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 03:01:16.463020 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 03:01:16.463260 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":250,"threads_started":1}
I20260619 03:01:16.465101 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.451669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.22.230.1:41207)
0619 03:01:16.452819 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.452829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.452856 (+ 27us) server_negotiation.cc:407] Connection header received
0619 03:01:16.452973 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.452981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.453215 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.453441 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:16.454178 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.456208 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.459887 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.461830 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.462009 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.462309 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.464388 (+ 2079us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.464627 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":829,"threads_started":1}
I20260619 03:01:16.475188 9879 tablet_service.cc:1467] Processing CreateTablet for tablet ab9dc047ad784eca8dc2fe380718db42 (DEFAULT_TABLE table=test-table [id=b771f0ce4fbd4d248f92fabfebccd885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:16.476338 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.452845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:33231 (local address 127.0.0.1:52752)
0619 03:01:16.464050 (+ 11205us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:16.464073 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:16.464203 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:16.470443 (+ 6240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:16.470451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:16.470487 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:16.470766 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.470778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.472525 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.472532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.474100 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.474111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.475453 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.475459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.475684 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.475721 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 03:01:16.475825 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 03:01:16.476013 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11055,"mutex_wait_us":27,"thread_start_us":16125,"threads_started":1}
I20260619 03:01:16.476403 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.453173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.22.230.2:33231)
0619 03:01:16.469778 (+ 16605us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.469789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.469821 (+ 32us) server_negotiation.cc:407] Connection header received
0619 03:01:16.469963 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.469970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.470218 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.470429 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:16.470922 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.472371 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.474244 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.475288 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.475443 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.475724 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.475863 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.476020 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16421,"thread_start_us":17523,"threads_started":1}
I20260619 03:01:16.476639 9879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9dc047ad784eca8dc2fe380718db42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:16.478885 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.454261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:39859 (local address 127.0.0.1:50084)
0619 03:01:16.469311 (+ 15050us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:16.469332 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:16.469470 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:16.470939 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:16.470946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:16.470976 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:16.471496 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.471509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.473351 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.473355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.475451 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.475465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.478085 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.478091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.478242 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.478271 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 03:01:16.478361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 03:01:16.478525 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14881,"mutex_wait_us":1,"thread_start_us":16957,"threads_started":1}
I20260619 03:01:16.479720 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.454967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.22.230.3:39859)
0619 03:01:16.470186 (+ 15219us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.470193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.470224 (+ 31us) server_negotiation.cc:407] Connection header received
0619 03:01:16.470319 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.470328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.470544 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.470780 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:16.471736 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.473191 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.475694 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.477594 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.477714 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.477964 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.479160 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.479310 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15068,"thread_start_us":14921,"threads_started":1}
I20260619 03:01:16.479449 10010 tablet_service.cc:1467] Processing CreateTablet for tablet ab9dc047ad784eca8dc2fe380718db42 (DEFAULT_TABLE table=test-table [id=b771f0ce4fbd4d248f92fabfebccd885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:16.480451 10010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9dc047ad784eca8dc2fe380718db42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:16.483394 10140 tablet_service.cc:1467] Processing CreateTablet for tablet ab9dc047ad784eca8dc2fe380718db42 (DEFAULT_TABLE table=test-table [id=b771f0ce4fbd4d248f92fabfebccd885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:16.484691 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9dc047ad784eca8dc2fe380718db42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:16.488191 10223 tablet_bootstrap.cc:492] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: Bootstrap starting.
I20260619 03:01:16.491353 10223 tablet_bootstrap.cc:654] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:16.492488 10223 log.cc:824] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:16.494319 10225 tablet_bootstrap.cc:492] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Bootstrap starting.
I20260619 03:01:16.495488 10223 tablet_bootstrap.cc:492] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: No bootstrap required, opened a new log
I20260619 03:01:16.495775 10223 ts_tablet_manager.cc:1397] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 03:01:16.497605 10225 tablet_bootstrap.cc:654] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:16.498704 10225 log.cc:824] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:16.499310 10226 tablet_bootstrap.cc:492] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Bootstrap starting.
I20260619 03:01:16.501849 10225 tablet_bootstrap.cc:492] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: No bootstrap required, opened a new log
I20260619 03:01:16.501880 10226 tablet_bootstrap.cc:654] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:16.502087 10225 ts_tablet_manager.cc:1397] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 03:01:16.502887 10226 log.cc:824] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:16.505523 10226 tablet_bootstrap.cc:492] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: No bootstrap required, opened a new log
I20260619 03:01:16.505759 10226 ts_tablet_manager.cc:1397] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 03:01:16.506551 10223 raft_consensus.cc:348] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.507040 10223 raft_consensus.cc:374] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:16.507133 10223 raft_consensus.cc:729] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d4e8a2ef4bd46b09f81e4eda941e835, State: Initialized, Role: FOLLOWER
I20260619 03:01:16.507694 10223 consensus_queue.cc:260] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.508689 9944 heartbeater.cc:503] Master 127.22.230.62:42089 was elected leader, sending a full tablet report...
I20260619 03:01:16.510607 10223 ts_tablet_manager.cc:1428] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 03:01:16.511665 10225 raft_consensus.cc:348] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.511976 10225 raft_consensus.cc:374] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:16.512089 10225 raft_consensus.cc:729] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc71048acf8a47d6b753541fa8f1cab2, State: Initialized, Role: FOLLOWER
I20260619 03:01:16.512658 10225 consensus_queue.cc:260] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.514195 10226 raft_consensus.cc:348] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.514492 10226 raft_consensus.cc:374] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:16.514636 10226 raft_consensus.cc:729] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad8f9b8c7f34f9cb16570b4230c451c, State: Initialized, Role: FOLLOWER
I20260619 03:01:16.515132 10226 consensus_queue.cc:260] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.515820 10075 heartbeater.cc:503] Master 127.22.230.62:42089 was elected leader, sending a full tablet report...
I20260619 03:01:16.516103 10205 heartbeater.cc:503] Master 127.22.230.62:42089 was elected leader, sending a full tablet report...
I20260619 03:01:16.516588 10226 ts_tablet_manager.cc:1428] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 03:01:16.517395 10225 ts_tablet_manager.cc:1428] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260619 03:01:16.561880 9945 tablet.cc:2369] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:01:16.651578 10206 tablet.cc:2369] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:01:16.732942 10076 tablet.cc:2369] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:01:16.891088 10229 raft_consensus.cc:482] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:16.891384 10229 raft_consensus.cc:504] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.893160 10229 leader_election.cc:288] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc71048acf8a47d6b753541fa8f1cab2 (127.22.230.2:33231), dad8f9b8c7f34f9cb16570b4230c451c (127.22.230.3:39859)
I20260619 03:01:16.899621 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.893588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:37847 (local address 127.22.230.3:39859)
0619 03:01:16.893795 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.893803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.894469 (+ 666us) server_negotiation.cc:407] Connection header received
0619 03:01:16.894656 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.894660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.894717 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.894829 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:16.895455 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.896169 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.897986 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.898717 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.898859 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.898935 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.899268 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.899395 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 03:01:16.899667 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.893422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:46991 (local address 127.22.230.2:33231)
0619 03:01:16.893630 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.893638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.894394 (+ 756us) server_negotiation.cc:407] Connection header received
0619 03:01:16.894605 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.894610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.894683 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.894748 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:16.895324 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.896061 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.898023 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.898729 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.898897 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.898959 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.899317 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.899447 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:01:16.899950 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.893367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:33231 (local address 127.22.230.1:46991)
0619 03:01:16.894278 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:16.894292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:16.894437 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:16.894794 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:16.894801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:16.894843 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:16.895186 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.895194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.896201 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.896209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.897824 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.897835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.898932 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.898938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.899150 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.899172 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 03:01:16.899277 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 03:01:16.899390 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":81,"thread_start_us":547,"threads_started":1}
I20260619 03:01:16.899972 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.893418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:39859 (local address 127.22.230.1:37847)
0619 03:01:16.894321 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:16.894334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:16.894506 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:16.894862 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:16.894869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:16.894896 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:16.895304 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.895314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.896294 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.896298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.897811 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:16.897822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.898880 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.898887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:16.899092 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.899140 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 03:01:16.899208 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 03:01:16.899332 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":48,"thread_start_us":616,"threads_started":1}
I20260619 03:01:16.901017 10160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9dc047ad784eca8dc2fe380718db42" candidate_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" is_pre_election: true
I20260619 03:01:16.901027 10030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9dc047ad784eca8dc2fe380718db42" candidate_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc71048acf8a47d6b753541fa8f1cab2" is_pre_election: true
I20260619 03:01:16.901525 10160 raft_consensus.cc:2454] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d4e8a2ef4bd46b09f81e4eda941e835 in term 0.
I20260619 03:01:16.901525 10030 raft_consensus.cc:2454] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d4e8a2ef4bd46b09f81e4eda941e835 in term 0.
I20260619 03:01:16.902084 9834 leader_election.cc:302] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4e8a2ef4bd46b09f81e4eda941e835, dad8f9b8c7f34f9cb16570b4230c451c; no voters:
I20260619 03:01:16.902382 10229 raft_consensus.cc:2781] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:01:16.902530 10229 raft_consensus.cc:482] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:01:16.902663 10229 raft_consensus.cc:3037] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:16.904588 10229 raft_consensus.cc:504] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.905076 10229 leader_election.cc:288] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [CANDIDATE]: Term 1 election: Requested vote from peers cc71048acf8a47d6b753541fa8f1cab2 (127.22.230.2:33231), dad8f9b8c7f34f9cb16570b4230c451c (127.22.230.3:39859)
I20260619 03:01:16.905458 10030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9dc047ad784eca8dc2fe380718db42" candidate_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc71048acf8a47d6b753541fa8f1cab2"
I20260619 03:01:16.905514 10160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9dc047ad784eca8dc2fe380718db42" candidate_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad8f9b8c7f34f9cb16570b4230c451c"
I20260619 03:01:16.905640 10030 raft_consensus.cc:3037] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:16.905742 10160 raft_consensus.cc:3037] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:16.907948 10030 raft_consensus.cc:2454] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4e8a2ef4bd46b09f81e4eda941e835 in term 1.
I20260619 03:01:16.908003 10160 raft_consensus.cc:2454] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d4e8a2ef4bd46b09f81e4eda941e835 in term 1.
I20260619 03:01:16.908353 9833 leader_election.cc:302] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d4e8a2ef4bd46b09f81e4eda941e835, cc71048acf8a47d6b753541fa8f1cab2; no voters:
I20260619 03:01:16.908633 10229 raft_consensus.cc:2781] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:16.909029 10229 raft_consensus.cc:686] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 1 LEADER]: Becoming Leader. State: Replica: 4d4e8a2ef4bd46b09f81e4eda941e835, State: Running, Role: LEADER
I20260619 03:01:16.909541 10229 consensus_queue.cc:237] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } }
I20260619 03:01:16.913782 9758 catalog_manager.cc:5495] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d4e8a2ef4bd46b09f81e4eda941e835 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d4e8a2ef4bd46b09f81e4eda941e835" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 41207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 } health_report { overall_health: UNKNOWN } } }
I20260619 03:01:16.933915 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.929279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.22.230.1:41207)
0619 03:01:16.929437 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.929441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.929456 (+ 15us) server_negotiation.cc:407] Connection header received
0619 03:01:16.929645 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.929648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.929730 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.929846 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:01:16.930165 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.930889 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.932120 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:16.932465 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:16.932574 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:16.932977 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:01:16.933396 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:01:16.933509 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:16.933614 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 03:01:16.933672 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 03:01:16.941385 10160 raft_consensus.cc:1261] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 1 FOLLOWER]: Refusing update from remote peer 4d4e8a2ef4bd46b09f81e4eda941e835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:16.941385 10030 raft_consensus.cc:1261] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 1 FOLLOWER]: Refusing update from remote peer 4d4e8a2ef4bd46b09f81e4eda941e835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:16.942226 10229 consensus_queue.cc:1035] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc71048acf8a47d6b753541fa8f1cab2" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 33231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:16.942477 10234 consensus_queue.cc:1035] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad8f9b8c7f34f9cb16570b4230c451c" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:16.964728 9757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:16.965005 9757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:16.968251 9757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0c03a1798634e68baf15e552a29d671: Sending DeleteTablet for 3 replicas of tablet ab9dc047ad784eca8dc2fe380718db42
I20260619 03:01:16.969168 10010 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9dc047ad784eca8dc2fe380718db42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:16 UTC) from {username='slave'} at 127.0.0.1:52752
I20260619 03:01:16.969257 10140 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9dc047ad784eca8dc2fe380718db42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:16 UTC) from {username='slave'} at 127.0.0.1:50084
I20260619 03:01:16.969291 9879 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9dc047ad784eca8dc2fe380718db42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:16 UTC) from {username='slave'} at 127.0.0.1:60640
I20260619 03:01:16.971202 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9817
I20260619 03:01:16.996996 10249 tablet_replica.cc:330] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: stopping tablet replica
I20260619 03:01:16.997536 10249 raft_consensus.cc:2229] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:16.998075 10249 raft_consensus.cc:2258] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:17.003865 10249 ts_tablet_manager.cc:1905] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:17.017081 10249 ts_tablet_manager.cc:1918] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:17.017302 10249 log.cc:1196] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/ab9dc047ad784eca8dc2fe380718db42
I20260619 03:01:17.017868 10249 ts_tablet_manager.cc:1939] T ab9dc047ad784eca8dc2fe380718db42 P dad8f9b8c7f34f9cb16570b4230c451c: Deleting consensus metadata
I20260619 03:01:17.019686 9744 catalog_manager.cc:4841] TS dad8f9b8c7f34f9cb16570b4230c451c (127.22.230.3:39859): tablet ab9dc047ad784eca8dc2fe380718db42 (table test-table [id=b771f0ce4fbd4d248f92fabfebccd885]) successfully deleted
I20260619 03:01:17.023109 10247 tablet_replica.cc:330] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: stopping tablet replica
I20260619 03:01:17.023586 10247 raft_consensus.cc:2229] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:17.024098 10247 raft_consensus.cc:2258] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:17.026391 10247 ts_tablet_manager.cc:1905] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:17.030679 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:16.972979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.22.230.1:41207)
0619 03:01:16.973216 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:16.973225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:16.999123 (+ 25898us) server_negotiation.cc:407] Connection header received
0619 03:01:16.999435 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:16.999444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:16.999587 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:16.999673 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:17.000463 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:17.001998 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:17.022761 (+ 20763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:17.023751 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:17.023920 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:17.025499 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:17.025511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:17.025527 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:17.026222 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:17.026282 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:17.026308 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:17.026378 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:17.026656 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 03:01:17.026767 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 03:01:17.031128 10248 tablet_replica.cc:330] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835: stopping tablet replica
I20260619 03:01:17.031323 9939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:17.034332 10247 ts_tablet_manager.cc:1918] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:17.034499 10247 log.cc:1196] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/ab9dc047ad784eca8dc2fe380718db42
I20260619 03:01:17.035068 10247 ts_tablet_manager.cc:1939] T ab9dc047ad784eca8dc2fe380718db42 P cc71048acf8a47d6b753541fa8f1cab2: Deleting consensus metadata
I20260619 03:01:17.036693 9743 catalog_manager.cc:4841] TS cc71048acf8a47d6b753541fa8f1cab2 (127.22.230.2:33231): tablet ab9dc047ad784eca8dc2fe380718db42 (table test-table [id=b771f0ce4fbd4d248f92fabfebccd885]) successfully deleted
I20260619 03:01:17.031494 10248 raft_consensus.cc:2229] T ab9dc047ad784eca8dc2fe380718db42 P 4d4e8a2ef4bd46b09f81e4eda941e835 [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:01:17.145520 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9817
I20260619 03:01:17.164664 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9948
W20260619 03:01:17.164664 9744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.230.1:41207 (error 108)
I20260619 03:01:17.166712 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.165853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60660)
0619 03:01:17.166353 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.166468 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":276,"threads_started":1}
W20260619 03:01:17.166980 9744 catalog_manager.cc:4584] TS 4d4e8a2ef4bd46b09f81e4eda941e835 (127.22.230.1:41207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab9dc047ad784eca8dc2fe380718db42: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
I20260619 03:01:17.171835 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.165492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.22.230.2:33231)
0619 03:01:17.165741 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:17.165748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:17.165941 (+ 193us) server_negotiation.cc:407] Connection header received
0619 03:01:17.166078 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:17.166087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:17.166217 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:17.166320 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:17.166834 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:17.167696 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:17.168920 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:17.169303 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:17.169433 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:17.170460 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:17.170469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:17.170481 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:17.171107 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:17.171160 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:17.171178 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:17.171299 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:17.171408 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 03:01:17.171505 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 03:01:17.172442 10070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:17.220400 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.219944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60666)
0619 03:01:17.220159 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.220251 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 03:01:17.221338 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.220983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60678)
0619 03:01:17.221121 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.221191 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 03:01:17.256238 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9948
I20260619 03:01:17.273303 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10078
I20260619 03:01:17.279907 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.273949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.22.230.3:39859)
0619 03:01:17.274105 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:17.274111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:17.274183 (+ 72us) server_negotiation.cc:407] Connection header received
0619 03:01:17.274379 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:17.274388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:17.274473 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:17.274564 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:17.274959 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:17.275573 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:17.276908 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:17.277319 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:17.277446 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:17.278498 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:17.278506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:17.278520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:17.279097 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:17.279141 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:17.279159 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:17.279348 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:17.279478 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 03:01:17.279575 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 03:01:17.280627 10200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:17.281785 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.281460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60690)
0619 03:01:17.281606 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.281673 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 03:01:17.283002 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.282593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60696)
0619 03:01:17.282767 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.282847 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 03:01:17.363034 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10078
I20260619 03:01:17.363401 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.362897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60698)
0619 03:01:17.363104 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.363218 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 03:01:17.364914 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:17.364471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:41207 (local address 127.0.0.1:60706)
0619 03:01:17.364662 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:17.364751 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:41207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 03:01:17.379135 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 9727
I20260619 03:01:17.380009 9788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:17.482558 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 9727
2026-06-19T03:01:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2951 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:17 dist-test-slave-2gms krb5kdc[10259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:17 dist-test-slave-2gms krb5kdc[10259](info): set up 2 sockets
Jun 19 03:01:17 dist-test-slave-2gms krb5kdc[10259](info): commencing operation
krb5kdc: starting...
W20260619 03:01:18.534778 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:18 dist-test-slave-2gms krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:18.759362 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35031
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:19.064893 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:19.065142 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:19.065224 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:19.072108 10272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:19.072196 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:19.072240 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:19.072293 10272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:19.072335 10272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:19.081424 10272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:35031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:19.082711 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:19.084014 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:19.095440 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:19.095476 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:19.095901 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:19.096128 10272 server_base.cc:1034] running on GCE node
I20260619 03:01:19.096856 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:19.098136 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:19.099385 10272 hybrid_clock.cc:648] HybridClock initialized: now 1781838079099349 us; error 25 us; skew 500 ppm
I20260619 03:01:19.099658 10272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:19 dist-test-slave-2gms krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:19.106530 10272 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:19.108165 10272 webserver.cc:460] Webserver started at http://127.22.230.62:40373/ using document root <none> and password file <none>
I20260619 03:01:19.108842 10272 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:19.108971 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:19.109270 10272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:19.112161 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "ac20273973da4725b0392d43411e3a46"
format_stamp: "Formatted at 2026-06-19 03:01:19 on dist-test-slave-2gms"
server_key: "bd859d5eae948d609066985cabc52d3c"
server_key_iv: "8205e5ff86a53da5a22097870150cfd7"
server_key_version: "encryptionkey@0"
I20260619 03:01:19.112879 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "ac20273973da4725b0392d43411e3a46"
format_stamp: "Formatted at 2026-06-19 03:01:19 on dist-test-slave-2gms"
server_key: "bd859d5eae948d609066985cabc52d3c"
server_key_iv: "8205e5ff86a53da5a22097870150cfd7"
server_key_version: "encryptionkey@0"
I20260619 03:01:19.117573 10272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 03:01:19.120807 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:19.121937 10272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:19.122073 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "ac20273973da4725b0392d43411e3a46"
format_stamp: "Formatted at 2026-06-19 03:01:19 on dist-test-slave-2gms"
server_key: "bd859d5eae948d609066985cabc52d3c"
server_key_iv: "8205e5ff86a53da5a22097870150cfd7"
server_key_version: "encryptionkey@0"
I20260619 03:01:19.122304 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:19.142514 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:19.145871 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:19.146083 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:19.162997 10272 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:35031
I20260619 03:01:19.163022 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:35031 every 8 connection(s)
I20260619 03:01:19.164088 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:19.167382 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:19.169394 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10272
I20260619 03:01:19.169582 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:19.169991 23448 external_mini_cluster.cc:1428] Setting key 97afb77484bea74aba4cb27681ef0716
I20260619 03:01:19.178133 10339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46: Bootstrap starting.
I20260619 03:01:19.181568 10339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:19.182854 10339 log.cc:824] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46: Log is configured to *not* fsync() on all Append() calls
Jun 19 03:01:19 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:19.185359 10339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46: No bootstrap required, opened a new log
I20260619 03:01:19.191123 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:19.171602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.22.230.62:35031)
0619 03:01:19.172314 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:19.172322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:19.172357 (+ 35us) server_negotiation.cc:407] Connection header received
0619 03:01:19.173782 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:19.173802 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:19.174144 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:19.174438 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:19.176096 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:19.177324 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:19.179003 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:19.179620 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:19.179741 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:19.184463 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:19.184470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:19.184480 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:19.184512 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:19.187494 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:19.188121 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:19.188126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:19.188131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:19.188297 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:19.188658 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:19.188661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:19.188663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:19.189075 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:19.189428 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:19.189857 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 03:01:19.190211 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":351,"threads_started":1}
I20260619 03:01:19.192993 10339 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac20273973da4725b0392d43411e3a46" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35031 } }
I20260619 03:01:19.193351 10339 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:19.193454 10339 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac20273973da4725b0392d43411e3a46, State: Initialized, Role: FOLLOWER
I20260619 03:01:19.194031 10339 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac20273973da4725b0392d43411e3a46" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35031 } }
I20260619 03:01:19.194257 10339 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:19.194383 10339 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:19.194543 10339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:19.196580 10339 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac20273973da4725b0392d43411e3a46" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35031 } }
I20260619 03:01:19.197013 10339 leader_election.cc:302] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac20273973da4725b0392d43411e3a46; no voters:
I20260619 03:01:19.197661 10339 leader_election.cc:288] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:19.197762 10344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:19.198477 10344 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [term 1 LEADER]: Becoming Leader. State: Replica: ac20273973da4725b0392d43411e3a46, State: Running, Role: LEADER
I20260619 03:01:19.199011 10344 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac20273973da4725b0392d43411e3a46" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35031 } }
I20260619 03:01:19.199482 10339 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:19.201397 10346 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac20273973da4725b0392d43411e3a46. Latest consensus state: current_term: 1 leader_uuid: "ac20273973da4725b0392d43411e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac20273973da4725b0392d43411e3a46" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35031 } } }
I20260619 03:01:19.201411 10345 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac20273973da4725b0392d43411e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac20273973da4725b0392d43411e3a46" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 35031 } } }
I20260619 03:01:19.201844 10345 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:19.202539 10346 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:19.203384 10349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:19.208403 10349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:19.216323 10349 catalog_manager.cc:1347] Generated new cluster ID: f8d237354ab8413985f828678c3523da
I20260619 03:01:19.216439 10349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:19.226120 10349 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:19.227334 10349 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:19.240495 10349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46: Generated new TSK 0
I20260619 03:01:19.241123 10349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
Entry for principal kudu/127.22.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab.
I20260619 03:01:19.311286 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35031
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:19.621459 10365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:19.621682 10365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:19.621822 10365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:19.628906 10365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:19.629011 10365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:19.629170 10365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:01:19.637944 10365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.1.keytab
--principal=kudu/127.22.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:19.638965 10365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:19.640084 10365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:19.654371 10370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:19.654405 10371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:19.654337 10373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:19.655085 10365 server_base.cc:1034] running on GCE node
I20260619 03:01:19.655682 10365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:19.656553 10365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:19.657737 10365 hybrid_clock.cc:648] HybridClock initialized: now 1781838079657695 us; error 40 us; skew 500 ppm
I20260619 03:01:19.658010 10365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:19 dist-test-slave-2gms krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:19.660953 10365 init.cc:377] Logged in from keytab as kudu/127.22.230.1@KRBTEST.COM (short username kudu)
I20260619 03:01:19.662227 10365 webserver.cc:460] Webserver started at http://127.22.230.1:37453/ using document root <none> and password file <none>
I20260619 03:01:19.662899 10365 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:19.663051 10365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:19.663364 10365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:19.666030 10365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "31c9851eb9d040c6a6ed82d61c0d84c4"
format_stamp: "Formatted at 2026-06-19 03:01:19 on dist-test-slave-2gms"
server_key: "74c52c7c7a25ab95584ffbc6a81dcb51"
server_key_iv: "489232afc34b0a60ecfbd56eb23a1537"
server_key_version: "encryptionkey@0"
I20260619 03:01:19.666812 10365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "31c9851eb9d040c6a6ed82d61c0d84c4"
format_stamp: "Formatted at 2026-06-19 03:01:19 on dist-test-slave-2gms"
server_key: "74c52c7c7a25ab95584ffbc6a81dcb51"
server_key_iv: "489232afc34b0a60ecfbd56eb23a1537"
server_key_version: "encryptionkey@0"
I20260619 03:01:19.671221 10365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 03:01:19.674165 10380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:19.675348 10365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:19.675513 10365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "31c9851eb9d040c6a6ed82d61c0d84c4"
format_stamp: "Formatted at 2026-06-19 03:01:19 on dist-test-slave-2gms"
server_key: "74c52c7c7a25ab95584ffbc6a81dcb51"
server_key_iv: "489232afc34b0a60ecfbd56eb23a1537"
server_key_version: "encryptionkey@0"
I20260619 03:01:19.675691 10365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:19.697018 10365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:19.701350 10365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:19.701606 10365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:19.702395 10365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:19.703835 10365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:19.703943 10365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:19.704067 10365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:19.704133 10365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:19.735159 10365 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:46005
I20260619 03:01:19.735210 10492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:46005 every 8 connection(s)
I20260619 03:01:19.736349 10365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:01:19.745261 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10365
I20260619 03:01:19.745440 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:01:19.745870 23448 external_mini_cluster.cc:1428] Setting key 5eef0656500f81bf7265d1ec8237e17b
Jun 19 03:01:19 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838079, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.1@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:19.755365 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:19.738677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:42243 (local address 127.22.230.62:35031)
0619 03:01:19.739085 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:19.739089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:19.739933 (+ 844us) server_negotiation.cc:407] Connection header received
0619 03:01:19.741626 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:19.741631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:19.741716 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:19.741835 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:19.743615 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:19.744264 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:19.745802 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:19.746262 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:19.746411 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:19.750084 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:19.750097 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:19.750100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:19.750130 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:19.752530 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:19.753231 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:19.753242 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:19.753246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:19.753358 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:19.753765 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:19.753770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:19.753775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:19.754018 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:19.754171 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:19.754847 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 03:01:19.755045 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":227,"threads_started":1}
I20260619 03:01:19.756170 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:19.739084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35031 (local address 127.22.230.1:42243)
0619 03:01:19.739787 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:19.739830 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:19.741384 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:19.742036 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:19.742042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:19.742731 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:19.743459 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:19.743475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:19.744403 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:19.744408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:19.745661 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:19.745673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:19.746468 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:19.746473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:19.746824 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:19.747498 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:19.747529 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:19.749886 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:19.752659 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:19.752666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:19.752679 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:19.753109 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:19.753466 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:19.753471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:19.753474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:19.753649 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:19.754141 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:19.754148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:19.754473 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 03:01:19.754733 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 03:01:19.755146 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":242,"threads_started":1}
I20260619 03:01:19.757529 10493 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35031
I20260619 03:01:19.757812 10493 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:19.758644 10493 heartbeater.cc:511] Master 127.22.230.62:35031 requested a full tablet report, sending...
I20260619 03:01:19.760519 10304 ts_manager.cc:194] Registered new tserver with Master: 31c9851eb9d040c6a6ed82d61c0d84c4 (127.22.230.1:46005)
I20260619 03:01:19.761914 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.1@KRBTEST.COM'} at 127.22.230.1:42243
WARNING: no policy specified for kudu/127.22.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
Entry for principal kudu/127.22.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab.
I20260619 03:01:19.799734 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35031
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:20.105948 10499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:20.106180 10499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:20.106272 10499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:20.113125 10499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:20.113219 10499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:20.113336 10499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:01:20.122465 10499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.2.keytab
--principal=kudu/127.22.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:20.123521 10499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:20.124658 10499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:20.136938 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:20.137007 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:20.136982 10505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:20.137846 10499 server_base.cc:1034] running on GCE node
I20260619 03:01:20.138324 10499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:20.139158 10499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:20.140348 10499 hybrid_clock.cc:648] HybridClock initialized: now 1781838080140290 us; error 53 us; skew 500 ppm
I20260619 03:01:20.140586 10499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:20 dist-test-slave-2gms krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:20.143400 10499 init.cc:377] Logged in from keytab as kudu/127.22.230.2@KRBTEST.COM (short username kudu)
I20260619 03:01:20.144774 10499 webserver.cc:460] Webserver started at http://127.22.230.2:36585/ using document root <none> and password file <none>
I20260619 03:01:20.145417 10499 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:20.145545 10499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:20.145848 10499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:20.148613 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "2eba8c96f014415eb17691aa2cf9a53d"
format_stamp: "Formatted at 2026-06-19 03:01:20 on dist-test-slave-2gms"
server_key: "fd19e11003fa61d7aca7569053211f3a"
server_key_iv: "19dc57dd48b2e8bf3d17271a8c6ac064"
server_key_version: "encryptionkey@0"
I20260619 03:01:20.149329 10499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "2eba8c96f014415eb17691aa2cf9a53d"
format_stamp: "Formatted at 2026-06-19 03:01:20 on dist-test-slave-2gms"
server_key: "fd19e11003fa61d7aca7569053211f3a"
server_key_iv: "19dc57dd48b2e8bf3d17271a8c6ac064"
server_key_version: "encryptionkey@0"
I20260619 03:01:20.154645 10499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 03:01:20.157702 10514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:20.158916 10499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:20.159077 10499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "2eba8c96f014415eb17691aa2cf9a53d"
format_stamp: "Formatted at 2026-06-19 03:01:20 on dist-test-slave-2gms"
server_key: "fd19e11003fa61d7aca7569053211f3a"
server_key_iv: "19dc57dd48b2e8bf3d17271a8c6ac064"
server_key_version: "encryptionkey@0"
I20260619 03:01:20.159257 10499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:20.172683 10499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:20.177341 10499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:20.177621 10499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:20.178653 10499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:20.180147 10499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:20.180230 10499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:20.180330 10499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:20.180380 10499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:20.214020 10499 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:42837
I20260619 03:01:20.214063 10626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:42837 every 8 connection(s)
I20260619 03:01:20.215214 10499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:01:20.220755 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10499
I20260619 03:01:20.220942 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:01:20.221477 23448 external_mini_cluster.cc:1428] Setting key d733cb3a29d04bfd868d7cba790b3510
Jun 19 03:01:20 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.2@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:20.234061 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.217535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:46379 (local address 127.22.230.62:35031)
0619 03:01:20.217726 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.217732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.218847 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 03:01:20.220429 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.220433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.220495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.220574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:20.222404 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.223161 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.224508 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.224977 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.225102 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.228809 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:20.228814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:20.228818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:20.228846 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:20.231337 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.231958 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.231964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.231967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.232092 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.232484 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.232490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.232493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.232737 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:20.232907 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.233478 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.233692 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 03:01:20.234937 10629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.218021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35031 (local address 127.22.230.2:46379)
0619 03:01:20.218708 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.218746 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.220202 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.220841 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.220853 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.221534 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:20.222221 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.222253 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.223296 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.223301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.224373 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.224383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.225184 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.225191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.225397 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.226064 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:20.226096 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:20.228610 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:20.231463 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:20.231470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:20.231482 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:20.231846 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:20.232197 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:20.232202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:20.232205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:20.232370 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:20.232876 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:20.232883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:20.233162 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.233399 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.233854 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":241,"threads_started":1}
I20260619 03:01:20.236260 10627 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35031
I20260619 03:01:20.236573 10627 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:20.237356 10627 heartbeater.cc:511] Master 127.22.230.62:35031 requested a full tablet report, sending...
I20260619 03:01:20.238807 10304 ts_manager.cc:194] Registered new tserver with Master: 2eba8c96f014415eb17691aa2cf9a53d (127.22.230.2:42837)
I20260619 03:01:20.239965 10304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.2@KRBTEST.COM'} at 127.22.230.2:46379
WARNING: no policy specified for kudu/127.22.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
Entry for principal kudu/127.22.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab.
I20260619 03:01:20.275141 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35031
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:20.577208 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:20.577442 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:20.577528 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:20.584484 10632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:20.584573 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:20.584671 10632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:01:20.593750 10632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.3.keytab
--principal=kudu/127.22.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:35031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:20.594780 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:20.595875 10632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:20.607930 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:20.607969 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:20.608011 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:20.612058 10632 server_base.cc:1034] running on GCE node
I20260619 03:01:20.612637 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:20.613481 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:20.614662 10632 hybrid_clock.cc:648] HybridClock initialized: now 1781838080614624 us; error 30 us; skew 500 ppm
I20260619 03:01:20.614954 10632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:20 dist-test-slave-2gms krb5kdc[10259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:20.617816 10632 init.cc:377] Logged in from keytab as kudu/127.22.230.3@KRBTEST.COM (short username kudu)
I20260619 03:01:20.619201 10632 webserver.cc:460] Webserver started at http://127.22.230.3:33199/ using document root <none> and password file <none>
I20260619 03:01:20.619819 10632 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:20.619956 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:20.620236 10632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:20.622885 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "eda132eb31cf4d88a2a486edd2a66ba9"
format_stamp: "Formatted at 2026-06-19 03:01:20 on dist-test-slave-2gms"
server_key: "c5211d10d9e4ab261016bf20e13d1c9f"
server_key_iv: "1c33de932091ba6ca3fe65525c7dec46"
server_key_version: "encryptionkey@0"
I20260619 03:01:20.623553 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "eda132eb31cf4d88a2a486edd2a66ba9"
format_stamp: "Formatted at 2026-06-19 03:01:20 on dist-test-slave-2gms"
server_key: "c5211d10d9e4ab261016bf20e13d1c9f"
server_key_iv: "1c33de932091ba6ca3fe65525c7dec46"
server_key_version: "encryptionkey@0"
I20260619 03:01:20.628247 10632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:01:20.631281 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:20.632423 10632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:20.632615 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "eda132eb31cf4d88a2a486edd2a66ba9"
format_stamp: "Formatted at 2026-06-19 03:01:20 on dist-test-slave-2gms"
server_key: "c5211d10d9e4ab261016bf20e13d1c9f"
server_key_iv: "1c33de932091ba6ca3fe65525c7dec46"
server_key_version: "encryptionkey@0"
I20260619 03:01:20.632795 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:20.649276 10632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:20.652987 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:20.653251 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:20.654079 10632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:20.655292 10632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:20.655396 10632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:20.655548 10632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:20.655619 10632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:20.688021 10632 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:45911
I20260619 03:01:20.688097 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:45911 every 8 connection(s)
I20260619 03:01:20.689164 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:01:20.694963 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10632
I20260619 03:01:20.695142 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:01:20.695654 23448 external_mini_cluster.cc:1428] Setting key ef0b373af3ce810c3a3c950acb1736b5
Jun 19 03:01:20 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838080, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.3@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:20.708137 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.691477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:60111 (local address 127.22.230.62:35031)
0619 03:01:20.691683 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.691690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.692773 (+ 1083us) server_negotiation.cc:407] Connection header received
0619 03:01:20.694448 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.694452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.694515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.694664 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:20.696494 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.697537 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.699352 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.699745 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.699837 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.702963 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:20.702967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:20.702970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:20.702997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:20.705470 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.706241 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.706247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.706250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.706389 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.706852 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.706855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.706857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.707110 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:20.707214 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.707749 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.707922 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 03:01:20.708967 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.691966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:35031 (local address 127.22.230.3:60111)
0619 03:01:20.692638 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.692673 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.694216 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.694886 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.694897 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.695595 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:20.696305 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.696330 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.697672 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.697675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.699223 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.699233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.699921 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.699926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.700147 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.700772 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:20.700795 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:20.702797 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:20.705596 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:20.705602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:20.705611 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:20.706127 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:20.706527 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:20.706531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:20.706533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:20.706759 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:20.707209 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:20.707214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:20.707468 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.707661 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.707976 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":259,"threads_started":1}
I20260619 03:01:20.710366 10760 heartbeater.cc:347] Connected to a master server at 127.22.230.62:35031
I20260619 03:01:20.710660 10760 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:20.711275 10760 heartbeater.cc:511] Master 127.22.230.62:35031 requested a full tablet report, sending...
I20260619 03:01:20.712503 10303 ts_manager.cc:194] Registered new tserver with Master: eda132eb31cf4d88a2a486edd2a66ba9 (127.22.230.3:45911)
I20260619 03:01:20.713282 10303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.230.3@KRBTEST.COM'} at 127.22.230.3:60111
I20260619 03:01:20.722095 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:01:20.737566 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.725818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.22.230.62:35031)
0619 03:01:20.726000 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.726003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.726507 (+ 504us) server_negotiation.cc:407] Connection header received
0619 03:01:20.726706 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.726709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.726762 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.726839 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:20.728081 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.728687 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.729836 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.730249 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.730343 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.732540 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:20.732545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:20.732547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:20.732575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:20.734804 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.735533 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.735537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.735539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.735665 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.736062 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.736068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.736071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.736388 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:20.736633 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.736995 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.737204 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 03:01:20.741087 10303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:01:20.747534 10303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:01:20.764621 10493 heartbeater.cc:503] Master 127.22.230.62:35031 was elected leader, sending a full tablet report...
I20260619 03:01:20.768967 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.758768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.22.230.3:45911)
0619 03:01:20.759611 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.759620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.760084 (+ 464us) server_negotiation.cc:407] Connection header received
0619 03:01:20.760360 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.760367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.760588 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.760732 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:20.761983 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.763322 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.766157 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.767597 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.767695 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.767927 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.768439 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.768613 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":485,"threads_started":1}
I20260619 03:01:20.769166 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.759023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:46005 (local address 127.0.0.1:56106)
0619 03:01:20.760004 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.760020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.760147 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.761226 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.761231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.761281 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:20.761817 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.761833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.764585 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.764590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.766754 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.766767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.768323 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.768332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.768485 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.768514 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.768604 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.768791 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":52,"spinlock_wait_cycles":5888,"thread_start_us":790,"threads_started":1}
I20260619 03:01:20.769344 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.759638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.22.230.1:46005)
0619 03:01:20.760711 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.760717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.760735 (+ 18us) server_negotiation.cc:407] Connection header received
0619 03:01:20.760796 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.760800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.761022 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.761611 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:20.762074 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.764437 (+ 2363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.767035 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.768112 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.768243 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.768452 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.768658 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.768820 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":790,"threads_started":1}
I20260619 03:01:20.769796 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.758629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:45911 (local address 127.0.0.1:36290)
0619 03:01:20.759951 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.759996 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.760178 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.761084 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.761091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.761118 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:20.761709 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.761733 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.763507 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.763510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.765972 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.765986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.768051 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.768055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.768170 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.768237 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.769293 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.769419 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":1,"thread_start_us":964,"threads_started":1}
I20260619 03:01:20.771241 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.760731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.22.230.2:42837)
0619 03:01:20.761246 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.761254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.762460 (+ 1206us) server_negotiation.cc:407] Connection header received
0619 03:01:20.762558 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.762584 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.762799 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.762971 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:20.763854 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.765371 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.767877 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.769241 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.769376 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.769564 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.770496 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.770672 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":264,"threads_started":1}
I20260619 03:01:20.771440 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.760333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:42837 (local address 127.0.0.1:50766)
0619 03:01:20.762194 (+ 1861us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.762216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.762323 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.763189 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.763193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.763216 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:20.763650 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.763662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.765541 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.765546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.767724 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.767737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.769723 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.769729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.769889 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.769929 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.770010 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.770117 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":1625,"threads_started":1}
I20260619 03:01:20.773149 10695 tablet_service.cc:1467] Processing CreateTablet for tablet d8356b1f0dc146cb86fd1cfd4b825547 (DEFAULT_TABLE table=test-table [id=4f943a436b3e42448706c3829ed22bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:20.773036 10428 tablet_service.cc:1467] Processing CreateTablet for tablet d8356b1f0dc146cb86fd1cfd4b825547 (DEFAULT_TABLE table=test-table [id=4f943a436b3e42448706c3829ed22bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:20.774262 10695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8356b1f0dc146cb86fd1cfd4b825547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:20.774381 10428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8356b1f0dc146cb86fd1cfd4b825547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:20.774724 10562 tablet_service.cc:1467] Processing CreateTablet for tablet d8356b1f0dc146cb86fd1cfd4b825547 (DEFAULT_TABLE table=test-table [id=4f943a436b3e42448706c3829ed22bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:20.775935 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8356b1f0dc146cb86fd1cfd4b825547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:20.784926 10778 tablet_bootstrap.cc:492] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Bootstrap starting.
I20260619 03:01:20.788017 10779 tablet_bootstrap.cc:492] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Bootstrap starting.
I20260619 03:01:20.789052 10778 tablet_bootstrap.cc:654] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:20.789121 10780 tablet_bootstrap.cc:492] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Bootstrap starting.
I20260619 03:01:20.790678 10778 log.cc:824] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:20.791359 10779 tablet_bootstrap.cc:654] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:20.791896 10780 tablet_bootstrap.cc:654] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:20.792387 10779 log.cc:824] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:20.792869 10780 log.cc:824] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:20.793628 10778 tablet_bootstrap.cc:492] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: No bootstrap required, opened a new log
I20260619 03:01:20.793879 10778 ts_tablet_manager.cc:1397] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260619 03:01:20.795154 10779 tablet_bootstrap.cc:492] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: No bootstrap required, opened a new log
I20260619 03:01:20.795377 10779 ts_tablet_manager.cc:1397] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 03:01:20.796247 10780 tablet_bootstrap.cc:492] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: No bootstrap required, opened a new log
I20260619 03:01:20.796496 10780 ts_tablet_manager.cc:1397] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 03:01:20.803678 10780 raft_consensus.cc:348] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.804008 10780 raft_consensus.cc:374] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:20.804102 10780 raft_consensus.cc:729] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eba8c96f014415eb17691aa2cf9a53d, State: Initialized, Role: FOLLOWER
I20260619 03:01:20.804312 10778 raft_consensus.cc:348] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.804689 10778 raft_consensus.cc:374] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:20.804800 10778 raft_consensus.cc:729] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda132eb31cf4d88a2a486edd2a66ba9, State: Initialized, Role: FOLLOWER
I20260619 03:01:20.804684 10780 consensus_queue.cc:260] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.805158 10779 raft_consensus.cc:348] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.805528 10779 raft_consensus.cc:374] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:20.805361 10778 consensus_queue.cc:260] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.805644 10779 raft_consensus.cc:729] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c9851eb9d040c6a6ed82d61c0d84c4, State: Initialized, Role: FOLLOWER
I20260619 03:01:20.806291 10779 consensus_queue.cc:260] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.806671 10760 heartbeater.cc:503] Master 127.22.230.62:35031 was elected leader, sending a full tablet report...
I20260619 03:01:20.807315 10778 ts_tablet_manager.cc:1428] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260619 03:01:20.808820 10779 ts_tablet_manager.cc:1428] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 03:01:20.809661 10627 heartbeater.cc:503] Master 127.22.230.62:35031 was elected leader, sending a full tablet report...
I20260619 03:01:20.810230 10780 ts_tablet_manager.cc:1428] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260619 03:01:20.835919 10784 raft_consensus.cc:482] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:20.836148 10784 raft_consensus.cc:504] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.837642 10784 leader_election.cc:288] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eda132eb31cf4d88a2a486edd2a66ba9 (127.22.230.3:45911), 31c9851eb9d040c6a6ed82d61c0d84c4 (127.22.230.1:46005)
I20260619 03:01:20.844214 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.838046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:40545 (local address 127.22.230.3:45911)
0619 03:01:20.838391 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.838399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.838946 (+ 547us) server_negotiation.cc:407] Connection header received
0619 03:01:20.839130 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.839135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.839197 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.839312 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:20.840018 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.840708 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.842461 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.843291 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.843417 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.843512 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.843853 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.843988 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260619 03:01:20.844226 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.838634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:34405 (local address 127.22.230.1:46005)
0619 03:01:20.838794 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.838800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.838946 (+ 146us) server_negotiation.cc:407] Connection header received
0619 03:01:20.839152 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.839156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.839226 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.839313 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:20.840139 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.840830 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.842528 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.843342 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.843524 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.843592 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.843908 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.844036 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 03:01:20.844696 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.837771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:45911 (local address 127.22.230.2:40545)
0619 03:01:20.838651 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.838838 (+ 187us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.838983 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.839342 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.839346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.839371 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:20.839858 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.839873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.840830 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.840836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.842309 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.842323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.843511 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.843517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.843697 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.843719 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.843790 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.843913 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":2,"spinlock_wait_cycles":17152,"thread_start_us":524,"threads_started":1}
I20260619 03:01:20.844702 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.838029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:46005 (local address 127.22.230.2:34405)
0619 03:01:20.838864 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:20.838880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:20.839014 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:20.839509 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:20.839518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:20.839545 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:20.839958 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.839971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.840961 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.840965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.842358 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:20.842396 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.843593 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.843598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:20.843729 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.843750 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 03:01:20.843872 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 03:01:20.843992 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":3,"thread_start_us":542,"threads_started":1}
I20260619 03:01:20.845615 10715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8356b1f0dc146cb86fd1cfd4b825547" candidate_uuid: "2eba8c96f014415eb17691aa2cf9a53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" is_pre_election: true
I20260619 03:01:20.845615 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8356b1f0dc146cb86fd1cfd4b825547" candidate_uuid: "2eba8c96f014415eb17691aa2cf9a53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" is_pre_election: true
I20260619 03:01:20.846118 10715 raft_consensus.cc:2454] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eba8c96f014415eb17691aa2cf9a53d in term 0.
I20260619 03:01:20.846118 10448 raft_consensus.cc:2454] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eba8c96f014415eb17691aa2cf9a53d in term 0.
I20260619 03:01:20.846771 10517 leader_election.cc:302] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eba8c96f014415eb17691aa2cf9a53d, eda132eb31cf4d88a2a486edd2a66ba9; no voters:
I20260619 03:01:20.847107 10784 raft_consensus.cc:2781] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:01:20.847250 10784 raft_consensus.cc:482] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:01:20.847352 10784 raft_consensus.cc:3037] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:20.849221 10784 raft_consensus.cc:504] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.849716 10784 leader_election.cc:288] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [CANDIDATE]: Term 1 election: Requested vote from peers eda132eb31cf4d88a2a486edd2a66ba9 (127.22.230.3:45911), 31c9851eb9d040c6a6ed82d61c0d84c4 (127.22.230.1:46005)
I20260619 03:01:20.850071 10715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8356b1f0dc146cb86fd1cfd4b825547" candidate_uuid: "2eba8c96f014415eb17691aa2cf9a53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda132eb31cf4d88a2a486edd2a66ba9"
I20260619 03:01:20.850169 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8356b1f0dc146cb86fd1cfd4b825547" candidate_uuid: "2eba8c96f014415eb17691aa2cf9a53d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4"
I20260619 03:01:20.850253 10715 raft_consensus.cc:3037] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:20.850333 10448 raft_consensus.cc:3037] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:20.853303 10448 raft_consensus.cc:2454] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eba8c96f014415eb17691aa2cf9a53d in term 1.
I20260619 03:01:20.853303 10715 raft_consensus.cc:2454] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eba8c96f014415eb17691aa2cf9a53d in term 1.
I20260619 03:01:20.853667 10517 leader_election.cc:302] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2eba8c96f014415eb17691aa2cf9a53d, 31c9851eb9d040c6a6ed82d61c0d84c4, eda132eb31cf4d88a2a486edd2a66ba9; no voters:
I20260619 03:01:20.853989 10784 raft_consensus.cc:2781] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:20.854380 10784 raft_consensus.cc:686] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 1 LEADER]: Becoming Leader. State: Replica: 2eba8c96f014415eb17691aa2cf9a53d, State: Running, Role: LEADER
I20260619 03:01:20.854854 10784 consensus_queue.cc:237] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } }
I20260619 03:01:20.858664 10304 catalog_manager.cc:5495] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eba8c96f014415eb17691aa2cf9a53d (127.22.230.2). New cstate: current_term: 1 leader_uuid: "2eba8c96f014415eb17691aa2cf9a53d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eba8c96f014415eb17691aa2cf9a53d" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 42837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 } health_report { overall_health: UNKNOWN } } }
I20260619 03:01:20.892181 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.887720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.22.230.2:42837)
0619 03:01:20.887899 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.887904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.887919 (+ 15us) server_negotiation.cc:407] Connection header received
0619 03:01:20.888050 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.888054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.888118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.888226 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:01:20.888512 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.889144 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.890359 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.890799 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.890903 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.891288 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:01:20.891683 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:01:20.891779 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.891848 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.891900 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 03:01:20.899240 10448 raft_consensus.cc:1261] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 1 FOLLOWER]: Refusing update from remote peer 2eba8c96f014415eb17691aa2cf9a53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:20.899384 10715 raft_consensus.cc:1261] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 1 FOLLOWER]: Refusing update from remote peer 2eba8c96f014415eb17691aa2cf9a53d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:20.900238 10784 consensus_queue.cc:1035] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c9851eb9d040c6a6ed82d61c0d84c4" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:20.900552 10789 consensus_queue.cc:1035] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eda132eb31cf4d88a2a486edd2a66ba9" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 45911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:20.911280 10792 mvcc.cc:205] Tried to move back new op lower bound from 7298408779354882048 to 7298408779184775168. Current Snapshot: MvccSnapshot[applied={T|T < 7298408779354882048}]
I20260619 03:01:20.922194 10304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:20.922433 10304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:20.925940 10304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac20273973da4725b0392d43411e3a46: Sending DeleteTablet for 3 replicas of tablet d8356b1f0dc146cb86fd1cfd4b825547
I20260619 03:01:20.927068 10695 tablet_service.cc:1514] Processing DeleteTablet for tablet d8356b1f0dc146cb86fd1cfd4b825547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:20 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:36290
I20260619 03:01:20.927691 10428 tablet_service.cc:1514] Processing DeleteTablet for tablet d8356b1f0dc146cb86fd1cfd4b825547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:20 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:56106
I20260619 03:01:20.927886 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10365
I20260619 03:01:20.928278 10562 tablet_service.cc:1514] Processing DeleteTablet for tablet d8356b1f0dc146cb86fd1cfd4b825547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:20 UTC) from {username='kudu', principal='kudu/127.22.230.62@KRBTEST.COM'} at 127.0.0.1:50766
I20260619 03:01:20.929747 10801 tablet_replica.cc:330] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: stopping tablet replica
I20260619 03:01:20.930271 10801 raft_consensus.cc:2229] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:20.930850 10801 raft_consensus.cc:2258] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:20.933146 10801 ts_tablet_manager.cc:1905] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:20.936086 10802 tablet_replica.cc:330] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: stopping tablet replica
I20260619 03:01:20.937254 10804 tablet_replica.cc:330] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: stopping tablet replica
I20260619 03:01:20.937599 10804 raft_consensus.cc:2229] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:01:20.938146 10804 raft_consensus.cc:2258] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:20.938712 10802 raft_consensus.cc:2229] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:20.939227 10802 raft_consensus.cc:2258] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:20.941111 10802 ts_tablet_manager.cc:1905] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:20.942559 10804 ts_tablet_manager.cc:1905] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:20.943372 10801 ts_tablet_manager.cc:1918] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:20.943544 10801 log.cc:1196] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/d8356b1f0dc146cb86fd1cfd4b825547
I20260619 03:01:20.944087 10801 ts_tablet_manager.cc:1939] T d8356b1f0dc146cb86fd1cfd4b825547 P eda132eb31cf4d88a2a486edd2a66ba9: Deleting consensus metadata
I20260619 03:01:20.945714 10290 catalog_manager.cc:4841] TS eda132eb31cf4d88a2a486edd2a66ba9 (127.22.230.3:45911): tablet d8356b1f0dc146cb86fd1cfd4b825547 (table test-table [id=4f943a436b3e42448706c3829ed22bb5]) successfully deleted
Jun 19 03:01:20 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.1@KRBTEST.COM
I20260619 03:01:20.950678 10802 ts_tablet_manager.cc:1918] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:20.950892 10802 log.cc:1196] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-0/wal/wals/d8356b1f0dc146cb86fd1cfd4b825547
I20260619 03:01:20.951439 10802 ts_tablet_manager.cc:1939] T d8356b1f0dc146cb86fd1cfd4b825547 P 31c9851eb9d040c6a6ed82d61c0d84c4: Deleting consensus metadata
I20260619 03:01:20.952356 10804 ts_tablet_manager.cc:1918] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:20.952527 10804 log.cc:1196] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/d8356b1f0dc146cb86fd1cfd4b825547
I20260619 03:01:20.953061 10804 ts_tablet_manager.cc:1939] T d8356b1f0dc146cb86fd1cfd4b825547 P 2eba8c96f014415eb17691aa2cf9a53d: Deleting consensus metadata
I20260619 03:01:20.953063 10288 catalog_manager.cc:4841] TS 31c9851eb9d040c6a6ed82d61c0d84c4 (127.22.230.1:46005): tablet d8356b1f0dc146cb86fd1cfd4b825547 (table test-table [id=4f943a436b3e42448706c3829ed22bb5]) successfully deleted
I20260619 03:01:20.954488 10290 catalog_manager.cc:4841] TS 2eba8c96f014415eb17691aa2cf9a53d (127.22.230.2:42837): tablet d8356b1f0dc146cb86fd1cfd4b825547 (table test-table [id=4f943a436b3e42448706c3829ed22bb5]) successfully deleted
I20260619 03:01:20.954900 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:20.928841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.22.230.1:46005)
0619 03:01:20.928980 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:20.928985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:20.936624 (+ 7639us) server_negotiation.cc:407] Connection header received
0619 03:01:20.936805 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:20.936812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:20.936920 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:20.937041 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:20.938769 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.939878 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.941831 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:20.942393 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:20.942481 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:20.947920 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:20.947934 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:20.947948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:20.947999 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:20.951895 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.952736 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.952746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.952751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.952887 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:20.953321 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:20.953326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:20.953330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:20.953667 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:20.953850 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:20.954300 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 03:01:20.954504 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 03:01:20.955448 10488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:21.060606 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10365
I20260619 03:01:21.089501 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10499
Jun 19 03:01:21 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.2@KRBTEST.COM
I20260619 03:01:21.106727 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:21.090346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.22.230.2:42837)
0619 03:01:21.090562 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:21.090569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:21.090642 (+ 73us) server_negotiation.cc:407] Connection header received
0619 03:01:21.090867 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:21.090873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:21.090967 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:21.091099 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:21.092672 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:21.093477 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:21.095312 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:21.095802 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:21.095902 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:21.100856 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:21.100869 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:21.100888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:21.100926 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:21.103873 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:21.104580 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:21.104587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:21.104589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:21.104718 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:21.105106 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:21.105110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:21.105112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:21.105436 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:21.105768 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:21.106028 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 03:01:21.106247 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 03:01:21.107460 10622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:21.213867 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10499
I20260619 03:01:21.232466 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10632
Jun 19 03:01:21 dist-test-slave-2gms krb5kdc[10259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.3@KRBTEST.COM
I20260619 03:01:21.245628 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:21.233103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.22.230.3:45911)
0619 03:01:21.233262 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:21.233266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:21.233282 (+ 16us) server_negotiation.cc:407] Connection header received
0619 03:01:21.233402 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:21.233405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:21.233466 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:21.233543 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:21.234669 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:21.235332 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:21.236598 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:21.236987 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:21.237079 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:21.241071 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:21.241084 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:21.241098 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:21.241133 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:21.243539 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:21.244096 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:21.244102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:21.244105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:21.244228 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:21.244539 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:21.244542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:21.244544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:21.244819 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:21.245042 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:21.245267 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 03:01:21.245410 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 03:01:21.246270 10755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:21.331264 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10632
I20260619 03:01:21.347220 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10272
I20260619 03:01:21.348243 10334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:21.452868 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10272
2026-06-19T03:01:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T03:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:21Z Disabled control of system clock
I20260619 03:01:21.528512 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36061
--webserver_interface=127.22.230.62
--webserver_port=0
--builtin_ntp_servers=127.22.230.20:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 03:01:21.829593 10812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:21.829821 10812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:21.829891 10812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:21.836838 10812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:21.836930 10812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:21.836967 10812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:21.837023 10812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:21.837069 10812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:21.845091 10812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:36061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:21.846174 10812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:21.847826 10812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:21.859025 10820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:21.859090 10817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:21.859068 10818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:21.859822 10812 server_base.cc:1034] running on GCE node
I20260619 03:01:21.860519 10812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:21.861665 10812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:21.862847 10812 hybrid_clock.cc:648] HybridClock initialized: now 1781838081862811 us; error 26 us; skew 500 ppm
I20260619 03:01:21.863175 10812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:21.865294 10812 webserver.cc:460] Webserver started at http://127.22.230.62:41185/ using document root <none> and password file <none>
I20260619 03:01:21.865890 10812 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:21.866007 10812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:21.866394 10812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:21.869138 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "a3cdfcde2928434e93f2198b97b4cb93"
format_stamp: "Formatted at 2026-06-19 03:01:21 on dist-test-slave-2gms"
server_key: "5b106ad10d87c82d53aa9e78ab00506a"
server_key_iv: "9c25ba2358906db462b547957cae7fdc"
server_key_version: "encryptionkey@0"
I20260619 03:01:21.869884 10812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "a3cdfcde2928434e93f2198b97b4cb93"
format_stamp: "Formatted at 2026-06-19 03:01:21 on dist-test-slave-2gms"
server_key: "5b106ad10d87c82d53aa9e78ab00506a"
server_key_iv: "9c25ba2358906db462b547957cae7fdc"
server_key_version: "encryptionkey@0"
I20260619 03:01:21.874329 10812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 03:01:21.877375 10826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:21.878650 10812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 03:01:21.878808 10812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "a3cdfcde2928434e93f2198b97b4cb93"
format_stamp: "Formatted at 2026-06-19 03:01:21 on dist-test-slave-2gms"
server_key: "5b106ad10d87c82d53aa9e78ab00506a"
server_key_iv: "9c25ba2358906db462b547957cae7fdc"
server_key_version: "encryptionkey@0"
I20260619 03:01:21.879004 10812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:21.889724 10812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:21.890419 10812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:21.890743 10812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:21.908193 10812 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:36061
I20260619 03:01:21.908222 10877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:36061 every 8 connection(s)
I20260619 03:01:21.909354 10812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:21.912786 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:21.917347 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10812
I20260619 03:01:21.917511 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:21.917982 23448 external_mini_cluster.cc:1428] Setting key 713a40fb27ade2077980b452812a7a40
I20260619 03:01:21.922891 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93: Bootstrap starting.
I20260619 03:01:21.926120 10878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:21.927292 10878 log.cc:824] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:21.930184 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93: No bootstrap required, opened a new log
I20260619 03:01:21.930414 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:21.919622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.22.230.62:36061)
0619 03:01:21.920461 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:21.920470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:21.920509 (+ 39us) server_negotiation.cc:407] Connection header received
0619 03:01:21.921983 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:21.922002 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:21.922388 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:21.922737 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:21.923226 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:21.924396 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:21.926256 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:21.926967 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:21.927096 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:21.928242 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:21.928253 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:21.928271 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:21.928877 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:21.928959 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:21.928977 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:21.929056 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:21.929258 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 03:01:21.929545 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":326,"threads_started":1}
I20260619 03:01:21.936821 10878 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cdfcde2928434e93f2198b97b4cb93" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36061 } }
I20260619 03:01:21.937146 10878 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:21.937256 10878 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cdfcde2928434e93f2198b97b4cb93, State: Initialized, Role: FOLLOWER
I20260619 03:01:21.937827 10878 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cdfcde2928434e93f2198b97b4cb93" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36061 } }
I20260619 03:01:21.938041 10878 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:21.938175 10878 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:21.938333 10878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:21.940232 10878 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cdfcde2928434e93f2198b97b4cb93" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36061 } }
I20260619 03:01:21.940649 10878 leader_election.cc:302] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3cdfcde2928434e93f2198b97b4cb93; no voters:
I20260619 03:01:21.941190 10878 leader_election.cc:288] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:21.941301 10883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:21.942059 10883 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [term 1 LEADER]: Becoming Leader. State: Replica: a3cdfcde2928434e93f2198b97b4cb93, State: Running, Role: LEADER
I20260619 03:01:21.942631 10883 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cdfcde2928434e93f2198b97b4cb93" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36061 } }
I20260619 03:01:21.943220 10878 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:21.945423 10885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cdfcde2928434e93f2198b97b4cb93. Latest consensus state: current_term: 1 leader_uuid: "a3cdfcde2928434e93f2198b97b4cb93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cdfcde2928434e93f2198b97b4cb93" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36061 } } }
I20260619 03:01:21.945925 10885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:21.945583 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3cdfcde2928434e93f2198b97b4cb93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cdfcde2928434e93f2198b97b4cb93" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 36061 } } }
I20260619 03:01:21.946321 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:21.947749 10892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:21.952054 10892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:21.960530 10892 catalog_manager.cc:1347] Generated new cluster ID: 46cb8288e76642d396107117038e4a52
I20260619 03:01:21.960615 10892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:21.972065 10892 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:21.973608 10892 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:21.986601 10892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93: Generated new TSK 0
I20260619 03:01:21.987473 10892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:21.998991 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.1:0
--local_ip_for_outbound_sockets=127.22.230.1
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36061
--builtin_ntp_servers=127.22.230.20:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 03:01:22.321470 10902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:22.321753 10902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:22.321928 10902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:22.328464 10902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:22.328586 10902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:22.328739 10902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.1
I20260619 03:01:22.336916 10902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.230.1
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.1
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:22.338021 10902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:22.339597 10902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:22.352182 10907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:22.352173 10908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:22.352175 10910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:22.352952 10902 server_base.cc:1034] running on GCE node
I20260619 03:01:22.353523 10902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:22.354353 10902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:22.355595 10902 hybrid_clock.cc:648] HybridClock initialized: now 1781838082355553 us; error 30 us; skew 500 ppm
I20260619 03:01:22.355892 10902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:22.357889 10902 webserver.cc:460] Webserver started at http://127.22.230.1:36671/ using document root <none> and password file <none>
I20260619 03:01:22.358552 10902 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:22.358731 10902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:22.359043 10902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:22.361610 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data/instance:
uuid: "e010254b24054ed9ba0a7a1b1fe269a0"
format_stamp: "Formatted at 2026-06-19 03:01:22 on dist-test-slave-2gms"
server_key: "1dd3a183c86c356b1733cd758caad706"
server_key_iv: "0db1729cf26c588d8b283ac52f1ecbff"
server_key_version: "encryptionkey@0"
I20260619 03:01:22.362344 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance:
uuid: "e010254b24054ed9ba0a7a1b1fe269a0"
format_stamp: "Formatted at 2026-06-19 03:01:22 on dist-test-slave-2gms"
server_key: "1dd3a183c86c356b1733cd758caad706"
server_key_iv: "0db1729cf26c588d8b283ac52f1ecbff"
server_key_version: "encryptionkey@0"
I20260619 03:01:22.367070 10902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 03:01:22.370239 10916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:22.371461 10902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 03:01:22.371618 10902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal
uuid: "e010254b24054ed9ba0a7a1b1fe269a0"
format_stamp: "Formatted at 2026-06-19 03:01:22 on dist-test-slave-2gms"
server_key: "1dd3a183c86c356b1733cd758caad706"
server_key_iv: "0db1729cf26c588d8b283ac52f1ecbff"
server_key_version: "encryptionkey@0"
I20260619 03:01:22.371799 10902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:22.398555 10902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:22.399370 10902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:22.399647 10902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:22.400518 10902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:22.401774 10902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:22.401880 10902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:22.402005 10902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:22.402100 10902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:22.432471 10902 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.1:38697
I20260619 03:01:22.432564 11028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.1:38697 every 8 connection(s)
I20260619 03:01:22.433607 10902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/data/info.pb
I20260619 03:01:22.435017 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 10902
I20260619 03:01:22.435174 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-0/wal/instance
I20260619 03:01:22.435603 23448 external_mini_cluster.cc:1428] Setting key 37f98ba9e2461f413d19e75fa680fd2c
I20260619 03:01:22.438549 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.2:0
--local_ip_for_outbound_sockets=127.22.230.2
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36061
--builtin_ntp_servers=127.22.230.20:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 03:01:22.446254 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:22.436572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:33209 (local address 127.22.230.62:36061)
0619 03:01:22.437064 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:22.437070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:22.438156 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 03:01:22.439972 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:22.439977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:22.440053 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:22.440170 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:22.441163 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.441886 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.443488 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.443921 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.444054 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:22.445431 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:22.445437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:22.445442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:22.445462 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:22.445509 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:22.445521 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:22.445616 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:22.445822 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 03:01:22.445914 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":245,"threads_started":1}
I20260619 03:01:22.447450 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:22.437165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36061 (local address 127.22.230.1:33209)
0619 03:01:22.438014 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:22.438054 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:22.439708 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:22.440405 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:22.440413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:22.440494 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:22.441002 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:22.441018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.442047 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.442051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:22.443343 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:22.443355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.444134 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.444143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:22.444400 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:22.445142 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:22.445145 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:22.445179 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:22.445246 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:22.445616 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:22.445629 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:22.445639 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 03:01:22.445818 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 03:01:22.446168 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":314,"threads_started":1}
I20260619 03:01:22.448954 11029 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36061
I20260619 03:01:22.449210 11029 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:22.449839 11029 heartbeater.cc:511] Master 127.22.230.62:36061 requested a full tablet report, sending...
I20260619 03:01:22.451995 10843 ts_manager.cc:194] Registered new tserver with Master: e010254b24054ed9ba0a7a1b1fe269a0 (127.22.230.1:38697)
I20260619 03:01:22.454108 10843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.1:33209
W20260619 03:01:22.747829 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:22.748050 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:22.748121 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:22.754684 11033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:22.754758 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:22.754850 11033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.2
I20260619 03:01:22.763134 11033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.230.2
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.2
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:22.764199 11033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:22.765723 11033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:22.777966 11038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:22.777993 11039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:22.778220 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:22.778729 11033 server_base.cc:1034] running on GCE node
I20260619 03:01:22.779253 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:22.780046 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:22.781281 11033 hybrid_clock.cc:648] HybridClock initialized: now 1781838082781225 us; error 67 us; skew 500 ppm
I20260619 03:01:22.781582 11033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:22.783707 11033 webserver.cc:460] Webserver started at http://127.22.230.2:34697/ using document root <none> and password file <none>
I20260619 03:01:22.784300 11033 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:22.784441 11033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:22.784734 11033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:22.787376 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data/instance:
uuid: "06cd39f7867045419f9363cf713fbd88"
format_stamp: "Formatted at 2026-06-19 03:01:22 on dist-test-slave-2gms"
server_key: "434c5cde5ac105adb256b55aebbba731"
server_key_iv: "c102069bffbb0b5545f1b0ab2dcf6953"
server_key_version: "encryptionkey@0"
I20260619 03:01:22.788100 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance:
uuid: "06cd39f7867045419f9363cf713fbd88"
format_stamp: "Formatted at 2026-06-19 03:01:22 on dist-test-slave-2gms"
server_key: "434c5cde5ac105adb256b55aebbba731"
server_key_iv: "c102069bffbb0b5545f1b0ab2dcf6953"
server_key_version: "encryptionkey@0"
I20260619 03:01:22.792505 11033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:01:22.795679 11047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:22.796808 11033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:22.796972 11033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal
uuid: "06cd39f7867045419f9363cf713fbd88"
format_stamp: "Formatted at 2026-06-19 03:01:22 on dist-test-slave-2gms"
server_key: "434c5cde5ac105adb256b55aebbba731"
server_key_iv: "c102069bffbb0b5545f1b0ab2dcf6953"
server_key_version: "encryptionkey@0"
I20260619 03:01:22.797168 11033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:22.820429 11033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:22.821107 11033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:22.821393 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:22.822196 11033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:22.823414 11033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:22.823522 11033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:22.823639 11033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:22.823720 11033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:22.855003 11033 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.2:38437
I20260619 03:01:22.855049 11159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.2:38437 every 8 connection(s)
I20260619 03:01:22.856148 11033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/data/info.pb
I20260619 03:01:22.865091 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11033
I20260619 03:01:22.865263 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal/instance
I20260619 03:01:22.865700 23448 external_mini_cluster.cc:1428] Setting key 696676f470eb2f87987c9f70c1918d1b
I20260619 03:01:22.867370 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:22.858365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.2:59367 (local address 127.22.230.62:36061)
0619 03:01:22.858534 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:22.858538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:22.859735 (+ 1197us) server_negotiation.cc:407] Connection header received
0619 03:01:22.861311 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:22.861316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:22.861381 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:22.861460 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:22.862336 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.863078 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.864604 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.865198 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.865332 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:22.866659 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:22.866665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:22.866669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:22.866703 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:22.866740 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:22.866758 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:22.866848 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:22.867064 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 03:01:22.867135 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 03:01:22.868507 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:22.858836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36061 (local address 127.22.230.2:59367)
0619 03:01:22.859621 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:22.859652 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:22.861100 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:22.861633 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:22.861640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:22.861720 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:22.862200 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:22.862215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.863205 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.863208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:22.864372 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:22.864395 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:22.865424 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:22.865430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:22.865611 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:22.866315 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:22.866319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:22.866364 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:22.866471 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:22.866860 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:22.866870 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:22.866877 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 03:01:22.867051 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 03:01:22.867366 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":237,"threads_started":1}
I20260619 03:01:22.868801 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.230.3:0
--local_ip_for_outbound_sockets=127.22.230.3
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36061
--builtin_ntp_servers=127.22.230.20:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 03:01:22.869855 11160 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36061
I20260619 03:01:22.870165 11160 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:22.870976 11160 heartbeater.cc:511] Master 127.22.230.62:36061 requested a full tablet report, sending...
I20260619 03:01:22.872186 10843 ts_manager.cc:194] Registered new tserver with Master: 06cd39f7867045419f9363cf713fbd88 (127.22.230.2:38437)
I20260619 03:01:22.872920 10843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.2:59367
W20260619 03:01:23.179873 11163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:23.180102 11163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:23.180174 11163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:23.187368 11163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:23.187470 11163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:23.187634 11163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.230.3
I20260619 03:01:23.196091 11163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.230.3
--webserver_port=0
--tserver_master_addrs=127.22.230.62:36061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.230.3
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:23.197185 11163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:23.198931 11163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:23.210958 11169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:23.210987 11168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:23.211018 11171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:23.211615 11163 server_base.cc:1034] running on GCE node
I20260619 03:01:23.212141 11163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:23.212998 11163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:23.214205 11163 hybrid_clock.cc:648] HybridClock initialized: now 1781838083214157 us; error 37 us; skew 500 ppm
I20260619 03:01:23.214481 11163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 03:01:23.216585 11163 webserver.cc:460] Webserver started at http://127.22.230.3:44259/ using document root <none> and password file <none>
I20260619 03:01:23.217208 11163 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:23.217342 11163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:23.217645 11163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:23.220252 11163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data/instance:
uuid: "c92d37ef9a3f446cb719775d5b4ec5cf"
format_stamp: "Formatted at 2026-06-19 03:01:23 on dist-test-slave-2gms"
server_key: "32010494547489637a17c4fcc8f85bdf"
server_key_iv: "a444ac2d8e5e053fdd01d5c115cf216c"
server_key_version: "encryptionkey@0"
I20260619 03:01:23.220956 11163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance:
uuid: "c92d37ef9a3f446cb719775d5b4ec5cf"
format_stamp: "Formatted at 2026-06-19 03:01:23 on dist-test-slave-2gms"
server_key: "32010494547489637a17c4fcc8f85bdf"
server_key_iv: "a444ac2d8e5e053fdd01d5c115cf216c"
server_key_version: "encryptionkey@0"
I20260619 03:01:23.225575 11163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 03:01:23.228744 11177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:23.229817 11163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 03:01:23.229986 11163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal
uuid: "c92d37ef9a3f446cb719775d5b4ec5cf"
format_stamp: "Formatted at 2026-06-19 03:01:23 on dist-test-slave-2gms"
server_key: "32010494547489637a17c4fcc8f85bdf"
server_key_iv: "a444ac2d8e5e053fdd01d5c115cf216c"
server_key_version: "encryptionkey@0"
I20260619 03:01:23.230206 11163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:23.246181 11163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:23.246881 11163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:23.247159 11163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:23.248020 11163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 03:01:23.249221 11163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 03:01:23.249358 11163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:23.249476 11163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 03:01:23.249571 11163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:23.280185 11163 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.3:34045
I20260619 03:01:23.280259 11289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.3:34045 every 8 connection(s)
I20260619 03:01:23.281347 11163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/data/info.pb
I20260619 03:01:23.284638 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11163
I20260619 03:01:23.284793 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal/instance
I20260619 03:01:23.285307 23448 external_mini_cluster.cc:1428] Setting key 182b2ebe7e5ea349503deed6e2d271f5
I20260619 03:01:23.292936 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.283587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.3:60285 (local address 127.22.230.62:36061)
0619 03:01:23.283790 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.283795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.285261 (+ 1466us) server_negotiation.cc:407] Connection header received
0619 03:01:23.287215 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.287220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.287298 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.287413 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:23.288404 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.289114 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.290549 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.291011 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.291102 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.292265 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:23.292270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:23.292273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:23.292288 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:23.292319 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:23.292330 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:23.292406 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.292598 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.292701 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 03:01:23.293762 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.284147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:36061 (local address 127.22.230.3:60285)
0619 03:01:23.285091 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.285147 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:23.287006 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:23.287582 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:23.287589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:23.287674 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:23.288262 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.288278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.289233 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.289236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.290426 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.290437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.291176 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.291180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.291355 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.291944 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 03:01:23.291947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:23.291987 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 03:01:23.292071 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:23.292426 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:23.292432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:23.292437 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 03:01:23.292564 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 03:01:23.292803 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":281,"threads_started":1}
I20260619 03:01:23.294968 11290 heartbeater.cc:347] Connected to a master server at 127.22.230.62:36061
I20260619 03:01:23.295221 11290 heartbeater.cc:464] Registering TS with master...
I20260619 03:01:23.295886 11290 heartbeater.cc:511] Master 127.22.230.62:36061 requested a full tablet report, sending...
I20260619 03:01:23.297204 10843 ts_manager.cc:194] Registered new tserver with Master: c92d37ef9a3f446cb719775d5b4ec5cf (127.22.230.3:34045)
I20260619 03:01:23.297971 10843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.230.3:60285
I20260619 03:01:23.300133 23448 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 03:01:23.309238 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.304098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.22.230.62:36061)
0619 03:01:23.304235 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.304239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.304703 (+ 464us) server_negotiation.cc:407] Connection header received
0619 03:01:23.304825 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.304828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.304881 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.304954 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:23.305387 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.306003 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.307405 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.307770 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.307857 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.308785 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:23.308789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:23.308792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:23.308805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:23.308827 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:23.308837 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:23.308895 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.308999 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.309056 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 03:01:23.312563 10843 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 03:01:23.318781 10843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 03:01:23.341185 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.328969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:38437 (local address 127.0.0.1:54222)
0619 03:01:23.331116 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.331160 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:23.331420 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:23.332534 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:23.332544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:23.332595 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:23.333112 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.333127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.335654 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.335662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.338598 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.338607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.340329 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.340335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.340538 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.340589 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 03:01:23.340719 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 03:01:23.340883 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"thread_start_us":9422,"threads_started":1}
I20260619 03:01:23.341224 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.329289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.22.230.2:38437)
0619 03:01:23.329847 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.329858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.331264 (+ 1406us) server_negotiation.cc:407] Connection header received
0619 03:01:23.331657 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.331666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.331951 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.332123 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:23.333349 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.335494 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.338791 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.340153 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.340283 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.340555 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.340756 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.340904 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":751,"threads_started":1}
I20260619 03:01:23.345124 11095 tablet_service.cc:1467] Processing CreateTablet for tablet a0872c8d5e944348bcf87d63482e7986 (DEFAULT_TABLE table=test-table [id=bcad5c83120e42899aa83876b34c8f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:23.345933 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.330480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:34045 (local address 127.0.0.1:46698)
0619 03:01:23.338443 (+ 7963us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.338471 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:23.338606 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:23.339433 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:23.339439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:23.339462 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:23.339845 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.339856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.341463 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.341467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.343277 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.343289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.345197 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.345203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.345359 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.345406 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 03:01:23.345481 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 03:01:23.345593 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7855,"thread_start_us":7440,"threads_started":1}
I20260619 03:01:23.346324 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.329347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:52932)
0619 03:01:23.338147 (+ 8800us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.338160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:23.338297 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:23.339090 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:23.339094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:23.339118 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:23.339457 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.339465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.341103 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.341109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.343762 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.343775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.345606 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.345613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.345776 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.345808 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 03:01:23.345914 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 03:01:23.346024 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8676,"thread_start_us":1395,"threads_started":1}
I20260619 03:01:23.346508 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.329890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.22.230.1:38697)
0619 03:01:23.330440 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.330449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.338258 (+ 7809us) server_negotiation.cc:407] Connection header received
0619 03:01:23.338461 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.338468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.338683 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.338859 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:23.339603 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.340958 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.343947 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.345323 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.345455 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.345669 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.345971 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.346139 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":1415,"threads_started":1}
I20260619 03:01:23.346596 11095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0872c8d5e944348bcf87d63482e7986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:23.346756 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.331039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.22.230.3:34045)
0619 03:01:23.331883 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.331893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.338723 (+ 6830us) server_negotiation.cc:407] Connection header received
0619 03:01:23.338808 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.338827 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.339031 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.339184 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:23.340041 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.341344 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.343470 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.344692 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.344821 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.345064 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.346195 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.346346 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":474,"threads_started":1}
I20260619 03:01:23.349853 10964 tablet_service.cc:1467] Processing CreateTablet for tablet a0872c8d5e944348bcf87d63482e7986 (DEFAULT_TABLE table=test-table [id=bcad5c83120e42899aa83876b34c8f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:23.350174 11225 tablet_service.cc:1467] Processing CreateTablet for tablet a0872c8d5e944348bcf87d63482e7986 (DEFAULT_TABLE table=test-table [id=bcad5c83120e42899aa83876b34c8f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 03:01:23.351020 10964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0872c8d5e944348bcf87d63482e7986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:23.351292 11225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0872c8d5e944348bcf87d63482e7986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:23.360579 11308 tablet_bootstrap.cc:492] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Bootstrap starting.
I20260619 03:01:23.363271 11309 tablet_bootstrap.cc:492] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Bootstrap starting.
I20260619 03:01:23.364531 11308 tablet_bootstrap.cc:654] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:23.365501 11308 log.cc:824] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:23.367604 11309 tablet_bootstrap.cc:654] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:23.368094 11310 tablet_bootstrap.cc:492] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Bootstrap starting.
I20260619 03:01:23.368847 11309 log.cc:824] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:23.369514 11308 tablet_bootstrap.cc:492] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: No bootstrap required, opened a new log
I20260619 03:01:23.369782 11308 ts_tablet_manager.cc:1397] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 03:01:23.371605 11309 tablet_bootstrap.cc:492] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: No bootstrap required, opened a new log
I20260619 03:01:23.371747 11310 tablet_bootstrap.cc:654] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:23.371886 11309 ts_tablet_manager.cc:1397] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 03:01:23.372876 11310 log.cc:824] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:23.376045 11310 tablet_bootstrap.cc:492] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: No bootstrap required, opened a new log
I20260619 03:01:23.376289 11310 ts_tablet_manager.cc:1397] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 03:01:23.379711 11309 raft_consensus.cc:348] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.380048 11309 raft_consensus.cc:374] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:23.379873 11308 raft_consensus.cc:348] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.380138 11309 raft_consensus.cc:729] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92d37ef9a3f446cb719775d5b4ec5cf, State: Initialized, Role: FOLLOWER
I20260619 03:01:23.380264 11308 raft_consensus.cc:374] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:23.380369 11308 raft_consensus.cc:729] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cd39f7867045419f9363cf713fbd88, State: Initialized, Role: FOLLOWER
I20260619 03:01:23.380852 11309 consensus_queue.cc:260] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.380885 11308 consensus_queue.cc:260] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.381863 11290 heartbeater.cc:503] Master 127.22.230.62:36061 was elected leader, sending a full tablet report...
I20260619 03:01:23.382426 11309 ts_tablet_manager.cc:1428] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260619 03:01:23.383280 11160 heartbeater.cc:503] Master 127.22.230.62:36061 was elected leader, sending a full tablet report...
I20260619 03:01:23.383976 11308 ts_tablet_manager.cc:1428] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260619 03:01:23.388738 11310 raft_consensus.cc:348] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.389063 11310 raft_consensus.cc:374] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:23.389175 11310 raft_consensus.cc:729] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e010254b24054ed9ba0a7a1b1fe269a0, State: Initialized, Role: FOLLOWER
I20260619 03:01:23.389675 11310 consensus_queue.cc:260] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.390609 11029 heartbeater.cc:503] Master 127.22.230.62:36061 was elected leader, sending a full tablet report...
I20260619 03:01:23.391153 11310 ts_tablet_manager.cc:1428] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260619 03:01:23.435918 11030 tablet.cc:2369] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:01:23.472137 11316 raft_consensus.cc:482] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 03:01:23.472393 11316 raft_consensus.cc:504] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.473899 11316 leader_election.cc:288] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06cd39f7867045419f9363cf713fbd88 (127.22.230.2:38437), c92d37ef9a3f446cb719775d5b4ec5cf (127.22.230.3:34045)
I20260619 03:01:23.482635 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.474345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:54771 (local address 127.22.230.2:38437)
0619 03:01:23.474534 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.474541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.475251 (+ 710us) server_negotiation.cc:407] Connection header received
0619 03:01:23.475457 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.475463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.475539 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.475634 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:23.476382 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.477500 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.480379 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.481565 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.481717 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.481788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.482250 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.482382 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:01:23.482658 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.474415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.230.1:41663 (local address 127.22.230.3:34045)
0619 03:01:23.474573 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.474597 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.475250 (+ 653us) server_negotiation.cc:407] Connection header received
0619 03:01:23.475457 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.475463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.475559 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.475639 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 03:01:23.476476 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.477514 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.480379 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.481567 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.481741 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.481801 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.482250 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.482374 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 03:01:23.482923 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.474286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.3:34045 (local address 127.22.230.1:41663)
0619 03:01:23.475146 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.475161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:23.475285 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:23.475667 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:23.475672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:23.475710 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:23.476295 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.476307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.477630 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.477636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.480170 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.480184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.481794 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.481800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.482054 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.482121 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 03:01:23.482209 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 03:01:23.482345 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":313,"spinlock_wait_cycles":3840,"thread_start_us":469,"threads_started":1}
I20260619 03:01:23.483043 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.474256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.2:38437 (local address 127.22.230.1:54771)
0619 03:01:23.475122 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.475145 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:23.475332 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:23.475667 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:23.475672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:23.475704 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 03:01:23.476221 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.476241 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.477632 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.477636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.480170 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:23.480184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.481755 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.481765 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:23.482053 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.482118 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 03:01:23.482209 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 03:01:23.482345 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":246,"spinlock_wait_cycles":28288,"thread_start_us":522,"threads_started":1}
I20260619 03:01:23.484051 11115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0872c8d5e944348bcf87d63482e7986" candidate_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06cd39f7867045419f9363cf713fbd88" is_pre_election: true
I20260619 03:01:23.484045 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0872c8d5e944348bcf87d63482e7986" candidate_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" is_pre_election: true
I20260619 03:01:23.484552 11115 raft_consensus.cc:2454] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e010254b24054ed9ba0a7a1b1fe269a0 in term 0.
I20260619 03:01:23.484552 11245 raft_consensus.cc:2454] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e010254b24054ed9ba0a7a1b1fe269a0 in term 0.
I20260619 03:01:23.485173 10917 leader_election.cc:302] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92d37ef9a3f446cb719775d5b4ec5cf, e010254b24054ed9ba0a7a1b1fe269a0; no voters:
I20260619 03:01:23.485575 11316 raft_consensus.cc:2781] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 03:01:23.485724 11316 raft_consensus.cc:482] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 03:01:23.485828 11316 raft_consensus.cc:3037] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:23.488852 11316 raft_consensus.cc:504] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.489564 11316 leader_election.cc:288] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [CANDIDATE]: Term 1 election: Requested vote from peers 06cd39f7867045419f9363cf713fbd88 (127.22.230.2:38437), c92d37ef9a3f446cb719775d5b4ec5cf (127.22.230.3:34045)
I20260619 03:01:23.490098 11245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0872c8d5e944348bcf87d63482e7986" candidate_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf"
I20260619 03:01:23.490290 11245 raft_consensus.cc:3037] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:23.490500 11115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0872c8d5e944348bcf87d63482e7986" candidate_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06cd39f7867045419f9363cf713fbd88"
I20260619 03:01:23.490711 11115 raft_consensus.cc:3037] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:23.493376 11245 raft_consensus.cc:2454] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e010254b24054ed9ba0a7a1b1fe269a0 in term 1.
I20260619 03:01:23.493731 11115 raft_consensus.cc:2454] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e010254b24054ed9ba0a7a1b1fe269a0 in term 1.
I20260619 03:01:23.493724 10917 leader_election.cc:302] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c92d37ef9a3f446cb719775d5b4ec5cf, e010254b24054ed9ba0a7a1b1fe269a0; no voters:
I20260619 03:01:23.493983 11316 raft_consensus.cc:2781] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:23.494429 11316 raft_consensus.cc:686] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 1 LEADER]: Becoming Leader. State: Replica: e010254b24054ed9ba0a7a1b1fe269a0, State: Running, Role: LEADER
I20260619 03:01:23.495112 11316 consensus_queue.cc:237] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } }
I20260619 03:01:23.498973 10843 catalog_manager.cc:5495] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e010254b24054ed9ba0a7a1b1fe269a0 (127.22.230.1). New cstate: current_term: 1 leader_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e010254b24054ed9ba0a7a1b1fe269a0" member_type: VOTER last_known_addr { host: "127.22.230.1" port: 38697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 } health_report { overall_health: UNKNOWN } } }
W20260619 03:01:23.532935 11291 tablet.cc:2369] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:01:23.533735 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.529341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.22.230.1:38697)
0619 03:01:23.529485 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.529489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.529503 (+ 14us) server_negotiation.cc:407] Connection header received
0619 03:01:23.529643 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.529647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.529708 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.529829 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:01:23.530091 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.530756 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.531925 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.532268 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.532347 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.532772 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:01:23.533242 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:01:23.533374 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.533472 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.533523 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 03:01:23.540426 11245 raft_consensus.cc:1261] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 1 FOLLOWER]: Refusing update from remote peer e010254b24054ed9ba0a7a1b1fe269a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:23.540457 11115 raft_consensus.cc:1261] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 1 FOLLOWER]: Refusing update from remote peer e010254b24054ed9ba0a7a1b1fe269a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:23.541203 11319 consensus_queue.cc:1035] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92d37ef9a3f446cb719775d5b4ec5cf" member_type: VOTER last_known_addr { host: "127.22.230.3" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:23.541510 11316 consensus_queue.cc:1035] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06cd39f7867045419f9363cf713fbd88" member_type: VOTER last_known_addr { host: "127.22.230.2" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:23.563400 10842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:23.563686 10842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 03:01:23.567284 10842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3cdfcde2928434e93f2198b97b4cb93: Sending DeleteTablet for 3 replicas of tablet a0872c8d5e944348bcf87d63482e7986
I20260619 03:01:23.568245 11095 tablet_service.cc:1514] Processing DeleteTablet for tablet a0872c8d5e944348bcf87d63482e7986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:23 UTC) from {username='slave'} at 127.0.0.1:54222
I20260619 03:01:23.568346 10964 tablet_service.cc:1514] Processing DeleteTablet for tablet a0872c8d5e944348bcf87d63482e7986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:23 UTC) from {username='slave'} at 127.0.0.1:52932
I20260619 03:01:23.569319 11225 tablet_service.cc:1514] Processing DeleteTablet for tablet a0872c8d5e944348bcf87d63482e7986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 03:01:23 UTC) from {username='slave'} at 127.0.0.1:46698
I20260619 03:01:23.569380 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10902
I20260619 03:01:23.572480 11334 tablet_replica.cc:330] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: stopping tablet replica
I20260619 03:01:23.572912 11334 raft_consensus.cc:2229] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:23.573908 11334 raft_consensus.cc:2258] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:23.576139 11334 ts_tablet_manager.cc:1905] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:23.579130 11333 tablet_replica.cc:330] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: stopping tablet replica
I20260619 03:01:23.579473 11333 raft_consensus.cc:2229] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 03:01:23.579986 11333 raft_consensus.cc:2258] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:23.583541 11333 ts_tablet_manager.cc:1905] T a0872c8d5e944348bcf87d63482e7986 P e010254b24054ed9ba0a7a1b1fe269a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:23.583854 11334 ts_tablet_manager.cc:1918] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:23.584012 11334 log.cc:1196] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-2/wal/wals/a0872c8d5e944348bcf87d63482e7986
I20260619 03:01:23.584492 11334 ts_tablet_manager.cc:1939] T a0872c8d5e944348bcf87d63482e7986 P c92d37ef9a3f446cb719775d5b4ec5cf: Deleting consensus metadata
I20260619 03:01:23.585680 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.570466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.22.230.1:38697)
0619 03:01:23.570648 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.570654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.571857 (+ 1203us) server_negotiation.cc:407] Connection header received
0619 03:01:23.572049 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.572054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.572130 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.572235 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:23.578392 (+ 6157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.579539 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.581646 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.582381 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.582469 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.584117 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:23.584127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:23.584142 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:23.584876 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:23.584935 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:23.584958 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:23.585054 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.585251 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.585341 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 03:01:23.586054 10827 catalog_manager.cc:4841] TS c92d37ef9a3f446cb719775d5b4ec5cf (127.22.230.3:34045): tablet a0872c8d5e944348bcf87d63482e7986 (table test-table [id=bcad5c83120e42899aa83876b34c8f01]) successfully deleted
I20260619 03:01:23.586813 11024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:23.587463 11332 tablet_replica.cc:330] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: stopping tablet replica
I20260619 03:01:23.587898 11332 raft_consensus.cc:2229] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 03:01:23.588418 11332 raft_consensus.cc:2258] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 03:01:23.590493 11332 ts_tablet_manager.cc:1905] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 03:01:23.594887 11332 ts_tablet_manager.cc:1918] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 03:01:23.594990 11332 log.cc:1196] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Deleting WAL directory at /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781837774831470-23448-0/minicluster-data/ts-1/wal/wals/a0872c8d5e944348bcf87d63482e7986
I20260619 03:01:23.595467 11332 ts_tablet_manager.cc:1939] T a0872c8d5e944348bcf87d63482e7986 P 06cd39f7867045419f9363cf713fbd88: Deleting consensus metadata
I20260619 03:01:23.596678 10828 catalog_manager.cc:4841] TS 06cd39f7867045419f9363cf713fbd88 (127.22.230.2:38437): tablet a0872c8d5e944348bcf87d63482e7986 (table test-table [id=bcad5c83120e42899aa83876b34c8f01]) successfully deleted
I20260619 03:01:23.675212 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10902
I20260619 03:01:23.693449 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11033
W20260619 03:01:23.693468 10829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.230.1:38697 (error 108)
I20260619 03:01:23.695263 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.694762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:52950)
0619 03:01:23.694958 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.695093 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 03:01:23.695513 10829 catalog_manager.cc:4584] TS e010254b24054ed9ba0a7a1b1fe269a0 (127.22.230.1:38697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0872c8d5e944348bcf87d63482e7986: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
I20260619 03:01:23.700129 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.694220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.22.230.2:38437)
0619 03:01:23.694384 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.694389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.694457 (+ 68us) server_negotiation.cc:407] Connection header received
0619 03:01:23.694707 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.694711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.694826 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.694930 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:23.695310 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.696125 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.697366 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.697758 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.697845 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.698838 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:23.698846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:23.698857 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:23.699427 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:23.699471 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:23.699494 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:23.699576 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.699711 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.699788 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 03:01:23.700830 11155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:23.748761 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.748337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:52962)
0619 03:01:23.748527 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.748610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 03:01:23.749639 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.749301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:52966)
0619 03:01:23.749445 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.749517 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 03:01:23.783105 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11033
I20260619 03:01:23.799726 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11163
I20260619 03:01:23.806242 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.800417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.22.230.3:34045)
0619 03:01:23.800604 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:23.800612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:23.800631 (+ 19us) server_negotiation.cc:407] Connection header received
0619 03:01:23.800848 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:23.800852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:23.800934 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:23.801036 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:23.801441 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.802121 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.803398 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:23.803770 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:23.803899 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:23.804965 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:23.804974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:23.804988 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:23.805500 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:23.805542 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:23.805559 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:23.805692 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:23.805835 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 03:01:23.805960 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 03:01:23.807011 11285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:23.810437 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.809844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:52974)
0619 03:01:23.810081 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.810201 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 03:01:23.811512 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.811167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:52986)
0619 03:01:23.811299 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.811376 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 03:01:23.891342 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11163
I20260619 03:01:23.891984 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.891520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:53000)
0619 03:01:23.891706 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.891799 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 03:01:23.893127 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:23.892754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.1:38697 (local address 127.0.0.1:53006)
0619 03:01:23.892967 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:23.893031 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 03:01:23.909535 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 10812
I20260619 03:01:23.910549 10873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:24.016232 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 10812
2026-06-19T03:01:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2562 ms)
[----------] 6 tests from EncryptionPolicyTest (20231 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:24 dist-test-slave-2gms krb5kdc[11343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:24 dist-test-slave-2gms krb5kdc[11343](info): set up 2 sockets
Jun 19 03:01:24 dist-test-slave-2gms krb5kdc[11343](info): commencing operation
krb5kdc: starting...
W20260619 03:01:25.083036 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:25 dist-test-slave-2gms krb5kdc[11343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:25.317780 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:25.622601 11356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:25.622907 11356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:25.622992 11356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:25.629678 11356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:25.629774 11356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:25.629815 11356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:25.629868 11356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:25.629900 11356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:25.638651 11356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:25.639650 11356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:25.640789 11356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:25.651773 11364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:25.651842 11361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:25.651820 11362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:25.652616 11356 server_base.cc:1034] running on GCE node
I20260619 03:01:25.653314 11356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:25.654536 11356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:25.655812 11356 hybrid_clock.cc:648] HybridClock initialized: now 1781838085655759 us; error 41 us; skew 500 ppm
I20260619 03:01:25.656140 11356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:25 dist-test-slave-2gms krb5kdc[11343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:25.661815 11356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:25.663228 11356 webserver.cc:460] Webserver started at http://127.0.0.1:44631/ using document root <none> and password file <none>
I20260619 03:01:25.663825 11356 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:25.663950 11356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:25.664242 11356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:25.666982 11356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "56b21cd451e8477f99d1ec1d4e104a2d"
format_stamp: "Formatted at 2026-06-19 03:01:25 on dist-test-slave-2gms"
server_key: "7cb81a965be46194003b69a34e9dbb61"
server_key_iv: "9fa2a0568adee2f57165d7989cabbba4"
server_key_version: "encryptionkey@0"
I20260619 03:01:25.667716 11356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "56b21cd451e8477f99d1ec1d4e104a2d"
format_stamp: "Formatted at 2026-06-19 03:01:25 on dist-test-slave-2gms"
server_key: "7cb81a965be46194003b69a34e9dbb61"
server_key_iv: "9fa2a0568adee2f57165d7989cabbba4"
server_key_version: "encryptionkey@0"
I20260619 03:01:25.672417 11356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:01:25.675611 11371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:25.676816 11356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 03:01:25.677023 11356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "56b21cd451e8477f99d1ec1d4e104a2d"
format_stamp: "Formatted at 2026-06-19 03:01:25 on dist-test-slave-2gms"
server_key: "7cb81a965be46194003b69a34e9dbb61"
server_key_iv: "9fa2a0568adee2f57165d7989cabbba4"
server_key_version: "encryptionkey@0"
I20260619 03:01:25.677211 11356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:25.689177 11356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:25.692816 11356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:25.693071 11356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:25.711112 11356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41613
I20260619 03:01:25.711146 11422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41613 every 8 connection(s)
I20260619 03:01:25.712401 11356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:25.715776 11423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:25.718784 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11356
I20260619 03:01:25.719018 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:25.719801 23448 external_mini_cluster.cc:1428] Setting key 569230bc71ce4bbe2a11438964b7914b
I20260619 03:01:25.727922 11423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d: Bootstrap starting.
I20260619 03:01:25.731508 11423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:25.732769 11423 log.cc:824] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d: Log is configured to *not* fsync() on all Append() calls
Jun 19 03:01:25 dist-test-slave-2gms krb5kdc[11343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 03:01:25.735467 11423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d: No bootstrap required, opened a new log
I20260619 03:01:25.742026 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:25.721620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56333 (local address 127.0.0.1:41613)
0619 03:01:25.722330 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:25.722341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:25.722377 (+ 36us) server_negotiation.cc:407] Connection header received
0619 03:01:25.724005 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:25.724024 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:25.724395 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:25.724709 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:25.726388 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:25.727670 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:25.729425 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:25.730046 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:25.730162 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:25.735117 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:25.735124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:25.735136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:25.735170 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:25.738135 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:25.738994 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:25.739000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:25.739005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:25.739174 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:25.739626 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:25.739630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:25.739632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:25.740092 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:25.740444 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:25.740850 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 03:01:25.741134 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":340,"threads_started":1}
I20260619 03:01:25.742985 11423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41613 } }
I20260619 03:01:25.743319 11423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:25.743422 11423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b21cd451e8477f99d1ec1d4e104a2d, State: Initialized, Role: FOLLOWER
I20260619 03:01:25.743950 11423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41613 } }
I20260619 03:01:25.744246 11423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:25.744407 11423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:25.744614 11423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:25.745903 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:25.747093 11423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41613 } }
I20260619 03:01:25.747546 11423 leader_election.cc:302] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56b21cd451e8477f99d1ec1d4e104a2d; no voters:
I20260619 03:01:25.748184 11423 leader_election.cc:288] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:25.748278 11428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:25.751020 11428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [term 1 LEADER]: Becoming Leader. State: Replica: 56b21cd451e8477f99d1ec1d4e104a2d, State: Running, Role: LEADER
I20260619 03:01:25.751588 11428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41613 } }
I20260619 03:01:25.752108 11423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:25.754702 11431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56b21cd451e8477f99d1ec1d4e104a2d. Latest consensus state: current_term: 1 leader_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41613 } } }
I20260619 03:01:25.755139 11431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:25.755396 11430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b21cd451e8477f99d1ec1d4e104a2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41613 } } }
I20260619 03:01:25.756050 11430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:25.761986 11441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:25.767272 11441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:25.771858 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:25.750649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37659 (local address 127.0.0.1:41613)
0619 03:01:25.750780 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:25.750785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:25.754078 (+ 3293us) server_negotiation.cc:407] Connection header received
0619 03:01:25.754253 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:25.754258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:25.754339 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:25.754446 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:25.756576 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:25.758037 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:25.760222 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:25.761159 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:25.761311 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:25.764472 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:25.764478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:25.764482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:25.764518 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:25.768324 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:25.769268 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:25.769273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:25.769278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:25.769420 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:25.770001 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:25.770006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:25.770009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:25.770454 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:25.770716 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:25.771345 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 03:01:25.771508 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 03:01:25.776786 11441 catalog_manager.cc:1347] Generated new cluster ID: 9ba119a19b5740d78cc11c68d890c054
I20260619 03:01:25.776880 11441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:25.791271 11441 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:25.792680 11441 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:25.797919 11441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56b21cd451e8477f99d1ec1d4e104a2d: Generated new TSK 0
I20260619 03:01:25.798774 11441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:25.812539 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11356
I20260619 03:01:25.813642 11418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:25.900727 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11356
W20260619 03:01:25.918768 11341 connection.cc:535] client connection to 127.0.0.1:41613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41613): Connection reset by peer (error 104)
2026-06-19T03:01:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:25 dist-test-slave-2gms krb5kdc[11455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:25 dist-test-slave-2gms krb5kdc[11455](info): set up 2 sockets
Jun 19 03:01:25 dist-test-slave-2gms krb5kdc[11455](info): commencing operation
krb5kdc: starting...
W20260619 03:01:26.986968 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:27 dist-test-slave-2gms krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:27.236009 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:27.548699 11468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:27.548995 11468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:27.549166 11468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:27.556082 11468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:27.556237 11468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:27.556339 11468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:27.556437 11468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:27.556519 11468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:27.565610 11468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:27.566797 11468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:27.567991 11468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:27.579314 11473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:27.579376 11476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:27.579411 11474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:27.580178 11468 server_base.cc:1034] running on GCE node
I20260619 03:01:27.580797 11468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:27.581934 11468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:27.583145 11468 hybrid_clock.cc:648] HybridClock initialized: now 1781838087583096 us; error 34 us; skew 500 ppm
I20260619 03:01:27.583473 11468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:27 dist-test-slave-2gms krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:27.586405 11468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:27.587816 11468 webserver.cc:460] Webserver started at http://127.0.0.1:33689/ using document root <none> and password file <none>
I20260619 03:01:27.588382 11468 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:27.588476 11468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:27.588698 11468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:27.591463 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e"
format_stamp: "Formatted at 2026-06-19 03:01:27 on dist-test-slave-2gms"
server_key: "de1083e071c8bdffb19087c7c51e4701"
server_key_iv: "77489a272b21b44add7ba2f43ec0de64"
server_key_version: "encryptionkey@0"
I20260619 03:01:27.592137 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e"
format_stamp: "Formatted at 2026-06-19 03:01:27 on dist-test-slave-2gms"
server_key: "de1083e071c8bdffb19087c7c51e4701"
server_key_iv: "77489a272b21b44add7ba2f43ec0de64"
server_key_version: "encryptionkey@0"
I20260619 03:01:27.597025 11468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 03:01:27.600133 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:27.601300 11468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:01:27.601471 11468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e"
format_stamp: "Formatted at 2026-06-19 03:01:27 on dist-test-slave-2gms"
server_key: "de1083e071c8bdffb19087c7c51e4701"
server_key_iv: "77489a272b21b44add7ba2f43ec0de64"
server_key_version: "encryptionkey@0"
I20260619 03:01:27.601647 11468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:27.614392 11468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:27.617991 11468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:27.618266 11468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:27.636225 11468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36883
I20260619 03:01:27.636302 11534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36883 every 8 connection(s)
I20260619 03:01:27.637460 11468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:27.640786 11535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:27.646324 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11468
I20260619 03:01:27.646500 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:27.647065 23448 external_mini_cluster.cc:1428] Setting key f43aa9ca5be297d59bbaadedef346d2b
I20260619 03:01:27.650907 11535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e: Bootstrap starting.
I20260619 03:01:27.654546 11535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:27.655781 11535 log.cc:824] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:27.658664 11535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e: No bootstrap required, opened a new log
I20260619 03:01:27.662642 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:27.648960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45297 (local address 127.0.0.1:36883)
0619 03:01:27.649816 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:27.649831 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:27.650244 (+ 413us) server_negotiation.cc:407] Connection header received
0619 03:01:27.652646 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:27.652671 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:27.653124 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:27.653513 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:27.653976 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:27.655973 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:27.657600 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:27.658633 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:27.658899 (+ 266us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:27.659991 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:27.660002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:27.660021 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:27.660711 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:27.660804 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:27.660837 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:27.660996 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:27.661202 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 03:01:27.661352 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":402,"threads_started":1}
I20260619 03:01:27.666133 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:27.668032 11535 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36883 } }
I20260619 03:01:27.668413 11535 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:27.668510 11535 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3c52e3c0ba46a5ab66927a1f77d51e, State: Initialized, Role: FOLLOWER
I20260619 03:01:27.669109 11535 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36883 } }
I20260619 03:01:27.669446 11535 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:27.669629 11535 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:27.669838 11535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:27.672741 11535 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36883 } }
I20260619 03:01:27.673283 11535 leader_election.cc:302] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd3c52e3c0ba46a5ab66927a1f77d51e; no voters:
I20260619 03:01:27.673883 11535 leader_election.cc:288] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:27.674006 11545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:27.675733 11535 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:27.679404 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:27.671358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60633 (local address 127.0.0.1:36883)
0619 03:01:27.671544 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:27.671556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:27.671935 (+ 379us) server_negotiation.cc:407] Connection header received
0619 03:01:27.672037 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:27.672042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:27.672104 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:27.672199 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:27.672629 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:27.673746 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:27.675948 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:27.676649 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:27.676785 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:27.678212 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:27.678221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:27.678229 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:27.678255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:27.678308 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:27.678334 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:27.678481 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:27.678851 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 03:01:27.678929 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 03:01:27.680411 11545 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [term 1 LEADER]: Becoming Leader. State: Replica: dd3c52e3c0ba46a5ab66927a1f77d51e, State: Running, Role: LEADER
I20260619 03:01:27.680888 11545 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36883 } }
I20260619 03:01:27.683674 11547 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd3c52e3c0ba46a5ab66927a1f77d51e. Latest consensus state: current_term: 1 leader_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36883 } } }
I20260619 03:01:27.683652 11546 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd3c52e3c0ba46a5ab66927a1f77d51e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36883 } } }
I20260619 03:01:27.684084 11546 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:27.684084 11547 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:27.690517 11557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:27.695298 11557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:27.702679 11557 catalog_manager.cc:1347] Generated new cluster ID: b3f779441f9748779735c95d7111beb4
I20260619 03:01:27.702775 11557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:27.729348 11557 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:27.730445 11557 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:27.734853 11557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd3c52e3c0ba46a5ab66927a1f77d51e: Generated new TSK 0
I20260619 03:01:27.735592 11557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:27.740221 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11468
W20260619 03:01:27.741227 11530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45297
W20260619 03:01:27.741920 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:27.742089 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11468
W20260619 03:01:27.756604 11453 connection.cc:535] client connection to 127.0.0.1:36883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36883): Connection reset by peer (error 104)
2026-06-19T03:01:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1837 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:27 dist-test-slave-2gms krb5kdc[11566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:27 dist-test-slave-2gms krb5kdc[11566](info): set up 2 sockets
Jun 19 03:01:27 dist-test-slave-2gms krb5kdc[11566](info): commencing operation
krb5kdc: starting...
W20260619 03:01:28.842973 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:28 dist-test-slave-2gms krb5kdc[11566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:29.069427 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:29.379508 11579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:29.379739 11579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:29.379884 11579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:29.386546 11579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:29.386691 11579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:29.386765 11579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:29.386858 11579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:29.386912 11579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:29.395869 11579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:29.396886 11579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:29.398092 11579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:29.409307 11584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:29.409368 11587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:29.409401 11585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:29.410066 11579 server_base.cc:1034] running on GCE node
I20260619 03:01:29.410769 11579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:29.411921 11579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:29.413122 11579 hybrid_clock.cc:648] HybridClock initialized: now 1781838089413078 us; error 33 us; skew 500 ppm
I20260619 03:01:29.413458 11579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:29 dist-test-slave-2gms krb5kdc[11566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:29.416790 11579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:29.418076 11579 webserver.cc:460] Webserver started at http://127.0.0.1:40775/ using document root <none> and password file <none>
I20260619 03:01:29.418712 11579 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:29.418812 11579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:29.419050 11579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:29.421721 11579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "4a89cac5242c4d3597da078a003cc08b"
format_stamp: "Formatted at 2026-06-19 03:01:29 on dist-test-slave-2gms"
server_key: "e628199a1f73f6722535b9c1ccd8227d"
server_key_iv: "e93240c237e20ad8fa2b76d547d11d1e"
server_key_version: "encryptionkey@0"
I20260619 03:01:29.422436 11579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "4a89cac5242c4d3597da078a003cc08b"
format_stamp: "Formatted at 2026-06-19 03:01:29 on dist-test-slave-2gms"
server_key: "e628199a1f73f6722535b9c1ccd8227d"
server_key_iv: "e93240c237e20ad8fa2b76d547d11d1e"
server_key_version: "encryptionkey@0"
I20260619 03:01:29.427290 11579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:01:29.430399 11594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:29.431617 11579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:29.431736 11579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "4a89cac5242c4d3597da078a003cc08b"
format_stamp: "Formatted at 2026-06-19 03:01:29 on dist-test-slave-2gms"
server_key: "e628199a1f73f6722535b9c1ccd8227d"
server_key_iv: "e93240c237e20ad8fa2b76d547d11d1e"
server_key_version: "encryptionkey@0"
I20260619 03:01:29.431883 11579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:29.437105 11579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:29.441190 11579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:29.441427 11579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:29.458874 11579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40219
I20260619 03:01:29.458950 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40219 every 8 connection(s)
I20260619 03:01:29.460091 11579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:29.463359 11646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:29.468204 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11579
I20260619 03:01:29.468369 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:29.468778 23448 external_mini_cluster.cc:1428] Setting key cc0233b03559dc580f1f93ebe6f20857
I20260619 03:01:29.474727 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b: Bootstrap starting.
I20260619 03:01:29.477130 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:29.470348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57387 (local address 127.0.0.1:40219)
0619 03:01:29.471172 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:29.471186 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:29.471233 (+ 47us) server_negotiation.cc:407] Connection header received
0619 03:01:29.473542 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:29.473566 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:29.473948 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:29.474361 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:29.475336 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:29.475344 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:29.475359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:29.475374 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:29.475433 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:29.475457 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:29.475653 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:29.475778 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 03:01:29.475953 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":366,"threads_started":1}
I20260619 03:01:29.478399 11646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:29.479578 11646 log.cc:824] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:29.480684 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:29.482463 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b: No bootstrap required, opened a new log
I20260619 03:01:29.488034 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:29.485650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42347 (local address 127.0.0.1:40219)
0619 03:01:29.485822 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:29.485828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:29.486220 (+ 392us) server_negotiation.cc:407] Connection header received
0619 03:01:29.486344 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:29.486348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:29.486412 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:29.486522 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:29.487383 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:29.487389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:29.487395 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:29.487417 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:29.487464 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:29.487483 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:29.487592 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:29.487674 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 03:01:29.487723 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:01:29.491607 11646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a89cac5242c4d3597da078a003cc08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40219 } }
I20260619 03:01:29.491895 11646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:29.491966 11646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a89cac5242c4d3597da078a003cc08b, State: Initialized, Role: FOLLOWER
I20260619 03:01:29.492385 11646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a89cac5242c4d3597da078a003cc08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40219 } }
I20260619 03:01:29.492635 11646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:29.492731 11646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:29.492843 11646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:29.495219 11646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a89cac5242c4d3597da078a003cc08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40219 } }
I20260619 03:01:29.495606 11646 leader_election.cc:302] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a89cac5242c4d3597da078a003cc08b; no voters:
I20260619 03:01:29.496100 11646 leader_election.cc:288] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:29.496233 11656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:29.496917 11656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [term 1 LEADER]: Becoming Leader. State: Replica: 4a89cac5242c4d3597da078a003cc08b, State: Running, Role: LEADER
I20260619 03:01:29.497421 11656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a89cac5242c4d3597da078a003cc08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40219 } }
I20260619 03:01:29.497937 11646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:29.500304 11657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a89cac5242c4d3597da078a003cc08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a89cac5242c4d3597da078a003cc08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40219 } } }
I20260619 03:01:29.500722 11657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:29.500855 11658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a89cac5242c4d3597da078a003cc08b. Latest consensus state: current_term: 1 leader_uuid: "4a89cac5242c4d3597da078a003cc08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a89cac5242c4d3597da078a003cc08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40219 } } }
I20260619 03:01:29.501246 11658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:29.501711 11664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:29.516919 11664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:29.528820 11664 catalog_manager.cc:1347] Generated new cluster ID: 8526d1960f1d4877b0c85a36ddd0a58c
I20260619 03:01:29.528920 11664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:29.576757 11664 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:29.577133 11664 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:29.584131 11664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a89cac5242c4d3597da078a003cc08b: Generated new TSK 0
I20260619 03:01:29.584851 11664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:29.597149 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11579
W20260619 03:01:29.598074 11641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57387
W20260619 03:01:29.598732 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11579: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:29.598896 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11579
2026-06-19T03:01:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:29 dist-test-slave-2gms krb5kdc[11677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:29 dist-test-slave-2gms krb5kdc[11677](info): set up 2 sockets
Jun 19 03:01:29 dist-test-slave-2gms krb5kdc[11677](info): commencing operation
krb5kdc: starting...
W20260619 03:01:30.674490 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:30 dist-test-slave-2gms krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:30.911533 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:31.216161 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:31.216398 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:31.216567 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:31.223281 11690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:31.223379 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:31.223488 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:31.223587 11690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:31.223659 11690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:31.232612 11690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:31.233609 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:31.234783 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:31.246026 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:31.246009 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:31.246029 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:31.246773 11690 server_base.cc:1034] running on GCE node
I20260619 03:01:31.247525 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:31.248659 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:31.249845 11690 hybrid_clock.cc:648] HybridClock initialized: now 1781838091249807 us; error 34 us; skew 500 ppm
I20260619 03:01:31.250152 11690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:31 dist-test-slave-2gms krb5kdc[11677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:31.257457 11690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:31.258894 11690 webserver.cc:460] Webserver started at http://127.0.0.1:43005/ using document root <none> and password file <none>
I20260619 03:01:31.259486 11690 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:31.259624 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:31.259913 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:31.262475 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "7b992277a58041a880308aa5ca70caad"
format_stamp: "Formatted at 2026-06-19 03:01:31 on dist-test-slave-2gms"
server_key: "1f3ac5236fb873d0482ffd2805c8a629"
server_key_iv: "8fe11b00d80ac468cfa563e5b3faf811"
server_key_version: "encryptionkey@0"
I20260619 03:01:31.263235 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "7b992277a58041a880308aa5ca70caad"
format_stamp: "Formatted at 2026-06-19 03:01:31 on dist-test-slave-2gms"
server_key: "1f3ac5236fb873d0482ffd2805c8a629"
server_key_iv: "8fe11b00d80ac468cfa563e5b3faf811"
server_key_version: "encryptionkey@0"
I20260619 03:01:31.267810 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 03:01:31.270882 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:31.272058 11690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:31.272225 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "7b992277a58041a880308aa5ca70caad"
format_stamp: "Formatted at 2026-06-19 03:01:31 on dist-test-slave-2gms"
server_key: "1f3ac5236fb873d0482ffd2805c8a629"
server_key_iv: "8fe11b00d80ac468cfa563e5b3faf811"
server_key_version: "encryptionkey@0"
I20260619 03:01:31.272393 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:31.288419 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:31.291982 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:31.292227 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:31.309721 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35255
I20260619 03:01:31.309811 11756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35255 every 8 connection(s)
I20260619 03:01:31.311061 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:31.314392 11757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:31.321159 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11690
I20260619 03:01:31.321346 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:31.321853 23448 external_mini_cluster.cc:1428] Setting key 3510ef09459259fa6205d7022fe28c03
I20260619 03:01:31.324700 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad: Bootstrap starting.
I20260619 03:01:31.327642 11757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:31.328812 11757 log.cc:824] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:31.331574 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad: No bootstrap required, opened a new log
Jun 19 03:01:31 dist-test-slave-2gms krb5kdc[11677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 03:01:31.339872 11757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b992277a58041a880308aa5ca70caad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260619 03:01:31.340286 11757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:31.340410 11757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b992277a58041a880308aa5ca70caad, State: Initialized, Role: FOLLOWER
I20260619 03:01:31.341070 11757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b992277a58041a880308aa5ca70caad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260619 03:01:31.341344 11757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:31.341495 11757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:31.341677 11757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:31.343632 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:31.323587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37153 (local address 127.0.0.1:35255)
0619 03:01:31.324244 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:31.324254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:31.324310 (+ 56us) server_negotiation.cc:407] Connection header received
0619 03:01:31.325896 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:31.325919 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:31.326300 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:31.326730 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:31.328149 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:31.330080 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:31.331430 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:31.332147 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:31.332342 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:31.336599 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:31.336609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:31.336621 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:31.336660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:31.339596 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:31.340429 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:31.340441 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:31.340446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:31.340636 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:31.341051 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:31.341054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:31.341059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:31.341704 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:31.342018 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:31.342387 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 03:01:31.342761 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":312,"threads_started":1}
I20260619 03:01:31.344170 11757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b992277a58041a880308aa5ca70caad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260619 03:01:31.344635 11757 leader_election.cc:302] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b992277a58041a880308aa5ca70caad; no voters:
I20260619 03:01:31.345181 11757 leader_election.cc:288] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:31.345358 11762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:31.346837 11762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [term 1 LEADER]: Becoming Leader. State: Replica: 7b992277a58041a880308aa5ca70caad, State: Running, Role: LEADER
I20260619 03:01:31.347446 11757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:31.347385 11762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b992277a58041a880308aa5ca70caad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260619 03:01:31.347872 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:31.355218 11763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b992277a58041a880308aa5ca70caad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b992277a58041a880308aa5ca70caad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } } }
I20260619 03:01:31.355630 11763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:31.356683 11764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b992277a58041a880308aa5ca70caad. Latest consensus state: current_term: 1 leader_uuid: "7b992277a58041a880308aa5ca70caad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b992277a58041a880308aa5ca70caad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } } }
I20260619 03:01:31.357124 11764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:31.359103 11771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:31.364302 11771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:31.378510 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:31.360042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48869 (local address 127.0.0.1:35255)
0619 03:01:31.360204 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:31.360209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:31.360440 (+ 231us) server_negotiation.cc:407] Connection header received
0619 03:01:31.360597 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:31.360603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:31.360673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:31.360801 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:31.362699 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:31.363717 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:31.365525 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:31.366216 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:31.366308 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:31.369416 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:31.369423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:31.369427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:31.369472 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:31.373673 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:31.374763 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:31.374772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:31.374778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:31.374976 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:31.375559 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:31.375566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:31.375572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:31.376037 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:31.376364 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:31.377715 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0619 03:01:31.378016 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 03:01:31.380888 11771 catalog_manager.cc:1347] Generated new cluster ID: 584743312ce64290a6cb6005ee4af11d
I20260619 03:01:31.380982 11771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:31.390877 11771 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:31.391958 11771 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:31.398509 11771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b992277a58041a880308aa5ca70caad: Generated new TSK 0
I20260619 03:01:31.399344 11771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:31.406750 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11690
I20260619 03:01:31.407694 11752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:31.492879 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11690
2026-06-19T03:01:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:31 dist-test-slave-2gms krb5kdc[11789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:31 dist-test-slave-2gms krb5kdc[11789](info): set up 2 sockets
Jun 19 03:01:31 dist-test-slave-2gms krb5kdc[11789](info): commencing operation
krb5kdc: starting...
W20260619 03:01:32.562872 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:32 dist-test-slave-2gms krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:32.792773 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:33.101706 11802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:33.102018 11802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:33.102227 11802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:33.109020 11802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:33.109154 11802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:33.109275 11802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:33.109383 11802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:33.109467 11802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:33.118847 11802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:33.120009 11802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:33.121241 11802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:33.133033 11808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:33.133064 11807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:33.132984 11810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:33.133793 11802 server_base.cc:1034] running on GCE node
I20260619 03:01:33.134522 11802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:33.135834 11802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:33.137059 11802 hybrid_clock.cc:648] HybridClock initialized: now 1781838093137028 us; error 39 us; skew 500 ppm
I20260619 03:01:33.137389 11802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:33 dist-test-slave-2gms krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:33.140452 11802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:33.141842 11802 webserver.cc:460] Webserver started at http://127.0.0.1:37545/ using document root <none> and password file <none>
I20260619 03:01:33.142473 11802 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:33.142664 11802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:33.142966 11802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:33.145789 11802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "defa0645003a400ea96059d821e71ca8"
format_stamp: "Formatted at 2026-06-19 03:01:33 on dist-test-slave-2gms"
server_key: "05dc15fa76befa5d12eb1cc513501f88"
server_key_iv: "b39b16b091ef8262c95dcf4c922686ce"
server_key_version: "encryptionkey@0"
I20260619 03:01:33.146513 11802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "defa0645003a400ea96059d821e71ca8"
format_stamp: "Formatted at 2026-06-19 03:01:33 on dist-test-slave-2gms"
server_key: "05dc15fa76befa5d12eb1cc513501f88"
server_key_iv: "b39b16b091ef8262c95dcf4c922686ce"
server_key_version: "encryptionkey@0"
I20260619 03:01:33.151484 11802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 03:01:33.154559 11817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:33.155730 11802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:33.155908 11802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "defa0645003a400ea96059d821e71ca8"
format_stamp: "Formatted at 2026-06-19 03:01:33 on dist-test-slave-2gms"
server_key: "05dc15fa76befa5d12eb1cc513501f88"
server_key_iv: "b39b16b091ef8262c95dcf4c922686ce"
server_key_version: "encryptionkey@0"
I20260619 03:01:33.156088 11802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:33.173942 11802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:33.177561 11802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:33.177821 11802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:33.195339 11802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37617
I20260619 03:01:33.195411 11868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37617 every 8 connection(s)
I20260619 03:01:33.196581 11802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:33.200055 11869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:33.202503 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11802
I20260619 03:01:33.202696 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:33.203356 23448 external_mini_cluster.cc:1428] Setting key 2ff63fd05c94d07738c136ef397a35a2
I20260619 03:01:33.211154 11869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8: Bootstrap starting.
I20260619 03:01:33.213891 11869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:33.215070 11869 log.cc:824] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:33.215256 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:33.205008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53829 (local address 127.0.0.1:37617)
0619 03:01:33.205691 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:33.205701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:33.205738 (+ 37us) server_negotiation.cc:407] Connection header received
0619 03:01:33.207422 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:33.207442 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:33.207794 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:33.208183 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:33.208690 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:33.209892 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:33.211232 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:33.211908 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:33.212118 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:33.213155 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:33.213162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:33.213177 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:33.213631 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:33.213698 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:33.213720 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:33.213858 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:33.214070 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 03:01:33.214320 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":297,"threads_started":1}
I20260619 03:01:33.217648 11869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8: No bootstrap required, opened a new log
I20260619 03:01:33.218593 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:33.226174 11869 raft_consensus.cc:348] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defa0645003a400ea96059d821e71ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37617 } }
I20260619 03:01:33.226590 11869 raft_consensus.cc:374] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:33.226773 11869 raft_consensus.cc:729] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defa0645003a400ea96059d821e71ca8, State: Initialized, Role: FOLLOWER
I20260619 03:01:33.227303 11869 consensus_queue.cc:260] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defa0645003a400ea96059d821e71ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37617 } }
I20260619 03:01:33.227588 11869 raft_consensus.cc:388] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:33.227780 11869 raft_consensus.cc:482] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:33.227986 11869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:33.229310 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:33.222604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51399 (local address 127.0.0.1:37617)
0619 03:01:33.222784 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:33.222790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:33.223110 (+ 320us) server_negotiation.cc:407] Connection header received
0619 03:01:33.223330 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:33.223333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:33.223389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:33.223471 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:33.224038 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:33.224733 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:33.226700 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:33.227338 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:33.227428 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:33.228569 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:33.228575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:33.228581 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:33.228602 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:33.228647 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:33.228669 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:33.228748 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:33.228884 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 03:01:33.228988 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 03:01:33.230533 11869 raft_consensus.cc:504] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defa0645003a400ea96059d821e71ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37617 } }
I20260619 03:01:33.231069 11869 leader_election.cc:302] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: defa0645003a400ea96059d821e71ca8; no voters:
I20260619 03:01:33.231602 11869 leader_election.cc:288] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:33.231737 11879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:33.232409 11879 raft_consensus.cc:686] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [term 1 LEADER]: Becoming Leader. State: Replica: defa0645003a400ea96059d821e71ca8, State: Running, Role: LEADER
I20260619 03:01:33.232962 11879 consensus_queue.cc:237] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defa0645003a400ea96059d821e71ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37617 } }
I20260619 03:01:33.233497 11869 sys_catalog.cc:564] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:33.235364 11881 sys_catalog.cc:455] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader defa0645003a400ea96059d821e71ca8. Latest consensus state: current_term: 1 leader_uuid: "defa0645003a400ea96059d821e71ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defa0645003a400ea96059d821e71ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37617 } } }
I20260619 03:01:33.235702 11881 sys_catalog.cc:458] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:33.236222 11880 sys_catalog.cc:455] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "defa0645003a400ea96059d821e71ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "defa0645003a400ea96059d821e71ca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37617 } } }
I20260619 03:01:33.236562 11880 sys_catalog.cc:458] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:33.239292 11885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:33.244204 11885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:33.252786 11885 catalog_manager.cc:1347] Generated new cluster ID: 6a8731ce133d491d9772beb3214ab10b
I20260619 03:01:33.252905 11885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:33.282313 11885 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:33.283540 11885 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:33.291647 11885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P defa0645003a400ea96059d821e71ca8: Generated new TSK 0
I20260619 03:01:33.292527 11885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:33.306206 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11802
W20260619 03:01:33.307749 11864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53829
W20260619 03:01:33.308486 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11802: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:33.308630 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11802
2026-06-19T03:01:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1816 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:33 dist-test-slave-2gms krb5kdc[11901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:33 dist-test-slave-2gms krb5kdc[11901](info): set up 2 sockets
Jun 19 03:01:33 dist-test-slave-2gms krb5kdc[11901](info): commencing operation
krb5kdc: starting...
W20260619 03:01:34.378768 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:34 dist-test-slave-2gms krb5kdc[11901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:34.611093 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:34.911345 11914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:34.911638 11914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:34.911799 11914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:34.918632 11914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:34.918764 11914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:34.918864 11914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:34.918972 11914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:34.919054 11914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:34.927888 11914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:34.929010 11914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:34.930176 11914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:34.941318 11919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:34.941285 11920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:34.941356 11922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:34.942199 11914 server_base.cc:1034] running on GCE node
I20260619 03:01:34.942921 11914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:34.944234 11914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:34.945472 11914 hybrid_clock.cc:648] HybridClock initialized: now 1781838094945415 us; error 44 us; skew 500 ppm
I20260619 03:01:34.945824 11914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:34 dist-test-slave-2gms krb5kdc[11901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:34.948871 11914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:34.950210 11914 webserver.cc:460] Webserver started at http://127.0.0.1:41483/ using document root <none> and password file <none>
I20260619 03:01:34.950881 11914 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:34.951025 11914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:34.951314 11914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:34.953933 11914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "7c5bb34a6299483495f2843c3fc1e91f"
format_stamp: "Formatted at 2026-06-19 03:01:34 on dist-test-slave-2gms"
server_key: "32b6d03ce8ff6474f92d977dbf6febff"
server_key_iv: "18ec83199cff5aa0f8daaf0fb6f94e46"
server_key_version: "encryptionkey@0"
I20260619 03:01:34.954676 11914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "7c5bb34a6299483495f2843c3fc1e91f"
format_stamp: "Formatted at 2026-06-19 03:01:34 on dist-test-slave-2gms"
server_key: "32b6d03ce8ff6474f92d977dbf6febff"
server_key_iv: "18ec83199cff5aa0f8daaf0fb6f94e46"
server_key_version: "encryptionkey@0"
I20260619 03:01:34.959246 11914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:01:34.962419 11929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:34.963581 11914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:34.963748 11914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "7c5bb34a6299483495f2843c3fc1e91f"
format_stamp: "Formatted at 2026-06-19 03:01:34 on dist-test-slave-2gms"
server_key: "32b6d03ce8ff6474f92d977dbf6febff"
server_key_iv: "18ec83199cff5aa0f8daaf0fb6f94e46"
server_key_version: "encryptionkey@0"
I20260619 03:01:34.963949 11914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:34.969229 11914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:34.973115 11914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:34.973393 11914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:34.990276 11914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35789
I20260619 03:01:34.990360 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35789 every 8 connection(s)
I20260619 03:01:34.991513 11914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:34.994742 11981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:35.000689 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 11914
I20260619 03:01:35.000872 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:35.001420 23448 external_mini_cluster.cc:1428] Setting key 189cfa16c2d54e5ed307bd579545c1d5
I20260619 03:01:35.004951 11981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f: Bootstrap starting.
I20260619 03:01:35.008042 11981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:35.008715 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:35.003139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44965 (local address 127.0.0.1:35789)
0619 03:01:35.003851 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:35.003861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:35.003904 (+ 43us) server_negotiation.cc:407] Connection header received
0619 03:01:35.005481 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:35.005499 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:35.005811 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:35.006171 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:35.007221 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:35.007226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:35.007244 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:35.007260 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:35.007309 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:35.007328 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:35.007513 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:35.007639 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 03:01:35.007817 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":319,"threads_started":1}
I20260619 03:01:35.009083 11981 log.cc:824] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:35.011755 11981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f: No bootstrap required, opened a new log
I20260619 03:01:35.011997 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:35.018141 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:35.015845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55305 (local address 127.0.0.1:35789)
0619 03:01:35.016052 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:35.016057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:35.016314 (+ 257us) server_negotiation.cc:407] Connection header received
0619 03:01:35.016480 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:35.016485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:35.016541 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:35.016644 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:35.017502 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:35.017507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:35.017512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:35.017531 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:35.017574 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:35.017591 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:35.017747 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:35.017826 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 03:01:35.017879 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 03:01:35.019567 11981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5bb34a6299483495f2843c3fc1e91f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260619 03:01:35.019884 11981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:35.019990 11981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5bb34a6299483495f2843c3fc1e91f, State: Initialized, Role: FOLLOWER
I20260619 03:01:35.020521 11981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5bb34a6299483495f2843c3fc1e91f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260619 03:01:35.020807 11981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:35.020967 11981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:35.021176 11981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:35.023213 11981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5bb34a6299483495f2843c3fc1e91f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260619 03:01:35.023653 11981 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5bb34a6299483495f2843c3fc1e91f; no voters:
I20260619 03:01:35.024201 11981 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:35.024287 11991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:35.024981 11991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [term 1 LEADER]: Becoming Leader. State: Replica: 7c5bb34a6299483495f2843c3fc1e91f, State: Running, Role: LEADER
I20260619 03:01:35.025496 11991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5bb34a6299483495f2843c3fc1e91f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } }
I20260619 03:01:35.026106 11981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:35.028225 11992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c5bb34a6299483495f2843c3fc1e91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5bb34a6299483495f2843c3fc1e91f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } } }
I20260619 03:01:35.028571 11992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:35.029381 11993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5bb34a6299483495f2843c3fc1e91f. Latest consensus state: current_term: 1 leader_uuid: "7c5bb34a6299483495f2843c3fc1e91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5bb34a6299483495f2843c3fc1e91f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35789 } } }
I20260619 03:01:35.029675 11993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:35.030871 11995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:35.035758 11995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:35.044147 11995 catalog_manager.cc:1347] Generated new cluster ID: cf87115d3a2c48fb8fe17fc6a4d1e903
I20260619 03:01:35.044274 11995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:35.074590 11995 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:35.074987 11995 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:35.092288 11995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c5bb34a6299483495f2843c3fc1e91f: Generated new TSK 0
I20260619 03:01:35.092945 11995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:35.109388 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11914
W20260619 03:01:35.110442 11976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44965
W20260619 03:01:35.111114 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 11914: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:35.111259 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 11914
2026-06-19T03:01:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1802 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:35 dist-test-slave-2gms krb5kdc[12012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:35 dist-test-slave-2gms krb5kdc[12012](info): set up 2 sockets
Jun 19 03:01:35 dist-test-slave-2gms krb5kdc[12012](info): commencing operation
krb5kdc: starting...
W20260619 03:01:36.186654 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:36 dist-test-slave-2gms krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:36.412911 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:36.715090 12025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:36.715435 12025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:36.715600 12025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:36.722491 12025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:36.722651 12025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:36.722745 12025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:36.722872 12025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:36.722939 12025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:36.732174 12025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:36.733331 12025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:36.734612 12025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:36.745687 12033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:36.745625 12031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:36.745728 12030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:36.746382 12025 server_base.cc:1034] running on GCE node
I20260619 03:01:36.747181 12025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:36.748489 12025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:36.749718 12025 hybrid_clock.cc:648] HybridClock initialized: now 1781838096749685 us; error 45 us; skew 500 ppm
I20260619 03:01:36.750068 12025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:36 dist-test-slave-2gms krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:36.757180 12025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:36.758450 12025 webserver.cc:460] Webserver started at http://127.0.0.1:46437/ using document root <none> and password file <none>
I20260619 03:01:36.759078 12025 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:36.759172 12025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:36.759407 12025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:36.762154 12025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "2afc717067c54fcda20f6b65abc234fe"
format_stamp: "Formatted at 2026-06-19 03:01:36 on dist-test-slave-2gms"
server_key: "5674f72a6550aa70ef468bcaf6d788a5"
server_key_iv: "f45187acc86ad294f564e47e44b723dc"
server_key_version: "encryptionkey@0"
I20260619 03:01:36.762892 12025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "2afc717067c54fcda20f6b65abc234fe"
format_stamp: "Formatted at 2026-06-19 03:01:36 on dist-test-slave-2gms"
server_key: "5674f72a6550aa70ef468bcaf6d788a5"
server_key_iv: "f45187acc86ad294f564e47e44b723dc"
server_key_version: "encryptionkey@0"
I20260619 03:01:36.767632 12025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 03:01:36.770669 12040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:36.771904 12025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:36.772027 12025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "2afc717067c54fcda20f6b65abc234fe"
format_stamp: "Formatted at 2026-06-19 03:01:36 on dist-test-slave-2gms"
server_key: "5674f72a6550aa70ef468bcaf6d788a5"
server_key_iv: "f45187acc86ad294f564e47e44b723dc"
server_key_version: "encryptionkey@0"
I20260619 03:01:36.772169 12025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:36.810774 12025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:36.814467 12025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:36.814747 12025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:36.831923 12025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46493
I20260619 03:01:36.831964 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46493 every 8 connection(s)
I20260619 03:01:36.833110 12025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:36.836444 12092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:36.842993 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12025
I20260619 03:01:36.843159 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:36.843535 23448 external_mini_cluster.cc:1428] Setting key 7c5edd004f7a805ac56ca1e0dcfda28f
I20260619 03:01:36.845932 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe: Bootstrap starting.
I20260619 03:01:36.849283 12092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:36.850363 12092 log.cc:824] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:36.852835 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe: No bootstrap required, opened a new log
Jun 19 03:01:36 dist-test-slave-2gms krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 03:01:36.860323 12092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc717067c54fcda20f6b65abc234fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260619 03:01:36.860661 12092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:36.860762 12092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afc717067c54fcda20f6b65abc234fe, State: Initialized, Role: FOLLOWER
I20260619 03:01:36.861336 12092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc717067c54fcda20f6b65abc234fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260619 03:01:36.861636 12092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:36.861768 12092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:36.861934 12092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:36.864257 12092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc717067c54fcda20f6b65abc234fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260619 03:01:36.864516 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:36.845055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:60787 (local address 127.0.0.1:46493)
0619 03:01:36.845746 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:36.845756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:36.845791 (+ 35us) server_negotiation.cc:407] Connection header received
0619 03:01:36.847646 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:36.847666 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:36.848009 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:36.848289 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:36.849898 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:36.851286 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:36.852477 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:36.853179 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:36.853393 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:36.857375 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:36.857382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:36.857394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:36.857433 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:36.860341 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:36.861235 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:36.861242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:36.861249 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:36.861462 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:36.861989 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:36.861992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:36.861994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:36.862510 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:36.862895 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:36.863343 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 03:01:36.863614 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":323,"threads_started":1}
I20260619 03:01:36.864799 12092 leader_election.cc:302] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2afc717067c54fcda20f6b65abc234fe; no voters:
I20260619 03:01:36.865483 12092 leader_election.cc:288] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:36.865594 12097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:36.866860 12097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [term 1 LEADER]: Becoming Leader. State: Replica: 2afc717067c54fcda20f6b65abc234fe, State: Running, Role: LEADER
I20260619 03:01:36.867450 12097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc717067c54fcda20f6b65abc234fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260619 03:01:36.867826 12092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:36.868530 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:36.870142 12099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2afc717067c54fcda20f6b65abc234fe. Latest consensus state: current_term: 1 leader_uuid: "2afc717067c54fcda20f6b65abc234fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc717067c54fcda20f6b65abc234fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } } }
I20260619 03:01:36.870594 12099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:36.871984 12098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2afc717067c54fcda20f6b65abc234fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2afc717067c54fcda20f6b65abc234fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } } }
I20260619 03:01:36.872334 12098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:36.883963 12103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:36.889842 12103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:36.907078 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:36.886994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:41527 (local address 127.0.0.1:46493)
0619 03:01:36.890736 (+ 3742us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:36.890742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:36.890758 (+ 16us) server_negotiation.cc:407] Connection header received
0619 03:01:36.890813 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:36.890818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:36.890887 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:36.890996 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:36.892851 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:36.894001 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:36.895801 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:36.896468 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:36.896576 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:36.899727 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:36.899733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:36.899738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:36.899773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:36.903275 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:36.904650 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:36.904655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:36.904659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:36.904806 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:36.905468 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:36.905473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:36.905477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:36.905816 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:36.906102 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:36.906509 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 03:01:36.906731 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3623}
I20260619 03:01:36.907748 12103 catalog_manager.cc:1347] Generated new cluster ID: 8254cab1a7e74a62a62820499b3828c5
I20260619 03:01:36.907850 12103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:36.917382 12103 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:36.919008 12103 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:36.924667 12103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2afc717067c54fcda20f6b65abc234fe: Generated new TSK 0
I20260619 03:01:36.925312 12103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:36.933909 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12025
I20260619 03:01:36.934813 12087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:37.020718 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12025
W20260619 03:01:37.038128 12010 connection.cc:535] client connection to 127.0.0.1:46493 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46493): Connection reset by peer (error 104)
2026-06-19T03:01:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:37 dist-test-slave-2gms krb5kdc[12125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:37 dist-test-slave-2gms krb5kdc[12125](info): set up 2 sockets
Jun 19 03:01:37 dist-test-slave-2gms krb5kdc[12125](info): commencing operation
krb5kdc: starting...
W20260619 03:01:38.106920 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:38 dist-test-slave-2gms krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:38.335474 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:38.641706 12138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:38.641950 12138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:38.642071 12138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:38.649155 12138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:38.649274 12138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:38.649369 12138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:38.649472 12138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:38.649550 12138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:38.659219 12138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:38.660300 12138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:38.661490 12138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:38.672950 12143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:38.672921 12146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:38.673136 12144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:38.673686 12138 server_base.cc:1034] running on GCE node
I20260619 03:01:38.674324 12138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:38.675567 12138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:38.676748 12138 hybrid_clock.cc:648] HybridClock initialized: now 1781838098676687 us; error 51 us; skew 500 ppm
I20260619 03:01:38.677021 12138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:38 dist-test-slave-2gms krb5kdc[12125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:38.680022 12138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:38.681262 12138 webserver.cc:460] Webserver started at http://127.0.0.1:43333/ using document root <none> and password file <none>
I20260619 03:01:38.681828 12138 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:38.681919 12138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:38.682135 12138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:38.684850 12138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "6ad38e136676427c93b6557c20bfe553"
format_stamp: "Formatted at 2026-06-19 03:01:38 on dist-test-slave-2gms"
server_key: "a8fbebb2d0a3e8ed6f5bf456bf403f74"
server_key_iv: "6c3cdf4f6f6eca3f6dd1f6c224ce76d1"
server_key_version: "encryptionkey@0"
I20260619 03:01:38.685515 12138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "6ad38e136676427c93b6557c20bfe553"
format_stamp: "Formatted at 2026-06-19 03:01:38 on dist-test-slave-2gms"
server_key: "a8fbebb2d0a3e8ed6f5bf456bf403f74"
server_key_iv: "6c3cdf4f6f6eca3f6dd1f6c224ce76d1"
server_key_version: "encryptionkey@0"
I20260619 03:01:38.690090 12138 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260619 03:01:38.693214 12153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:38.694337 12138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 03:01:38.694453 12138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "6ad38e136676427c93b6557c20bfe553"
format_stamp: "Formatted at 2026-06-19 03:01:38 on dist-test-slave-2gms"
server_key: "a8fbebb2d0a3e8ed6f5bf456bf403f74"
server_key_iv: "6c3cdf4f6f6eca3f6dd1f6c224ce76d1"
server_key_version: "encryptionkey@0"
I20260619 03:01:38.694664 12138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:38.704026 12138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:38.707823 12138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:38.708047 12138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:38.724970 12138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33853
I20260619 03:01:38.725006 12204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33853 every 8 connection(s)
I20260619 03:01:38.726100 12138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:38.726856 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12138
I20260619 03:01:38.727035 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:38.727558 23448 external_mini_cluster.cc:1428] Setting key 82d1c198fa89c2c74571de7c956a155e
I20260619 03:01:38.730129 12205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:38.739984 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:38.729243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:48995 (local address 127.0.0.1:33853)
0619 03:01:38.730004 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:38.730014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:38.730050 (+ 36us) server_negotiation.cc:407] Connection header received
0619 03:01:38.731652 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:38.731671 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:38.732030 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:38.732321 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:38.732885 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:38.734183 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:38.735983 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:38.736559 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:38.736673 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:38.737906 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:38.737913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:38.737924 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:38.738403 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:38.738470 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:38.738492 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:38.738680 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:38.738977 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 03:01:38.739144 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":330,"threads_started":1}
I20260619 03:01:38.741568 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553: Bootstrap starting.
I20260619 03:01:38.743491 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:38.744973 12205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:38.746052 12205 log.cc:824] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:38.749111 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553: No bootstrap required, opened a new log
I20260619 03:01:38.755297 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:38.748522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:43633 (local address 127.0.0.1:33853)
0619 03:01:38.748720 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:38.748728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:38.749184 (+ 456us) server_negotiation.cc:407] Connection header received
0619 03:01:38.749322 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:38.749325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:38.749371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:38.749435 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:38.750174 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:38.751007 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:38.752912 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:38.753358 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:38.753439 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:38.754533 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:38.754537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:38.754540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:38.754557 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:38.754648 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:38.754662 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:38.754741 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:38.754944 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 03:01:38.754997 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 03:01:38.757994 12205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad38e136676427c93b6557c20bfe553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260619 03:01:38.758332 12205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:38.758442 12205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad38e136676427c93b6557c20bfe553, State: Initialized, Role: FOLLOWER
I20260619 03:01:38.759086 12205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad38e136676427c93b6557c20bfe553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260619 03:01:38.759333 12205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:38.759470 12205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:38.759634 12205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:38.761631 12205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad38e136676427c93b6557c20bfe553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260619 03:01:38.762120 12205 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad38e136676427c93b6557c20bfe553; no voters:
I20260619 03:01:38.762727 12205 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:38.762843 12215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:38.763509 12215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad38e136676427c93b6557c20bfe553, State: Running, Role: LEADER
I20260619 03:01:38.764060 12215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad38e136676427c93b6557c20bfe553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260619 03:01:38.764710 12205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:38.766933 12216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad38e136676427c93b6557c20bfe553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad38e136676427c93b6557c20bfe553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } } }
I20260619 03:01:38.767439 12216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:38.767331 12217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad38e136676427c93b6557c20bfe553. Latest consensus state: current_term: 1 leader_uuid: "6ad38e136676427c93b6557c20bfe553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad38e136676427c93b6557c20bfe553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } } }
I20260619 03:01:38.767625 12217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:38.768834 12222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:38.773536 12222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:38.780891 12222 catalog_manager.cc:1347] Generated new cluster ID: b9c5561c64614d8dbc377e441e685903
I20260619 03:01:38.781006 12222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:38.794370 12222 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:38.795523 12222 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:38.805449 12222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ad38e136676427c93b6557c20bfe553: Generated new TSK 0
I20260619 03:01:38.806102 12222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:38.815481 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12138
W20260619 03:01:38.816715 12200 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.210:48995
W20260619 03:01:38.817448 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12138: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:38.817597 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12138
W20260619 03:01:38.834702 12123 connection.cc:535] client connection to 127.0.0.1:33853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33853): Connection reset by peer (error 104)
2026-06-19T03:01:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1796 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:38 dist-test-slave-2gms krb5kdc[12236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:38 dist-test-slave-2gms krb5kdc[12236](info): set up 2 sockets
Jun 19 03:01:38 dist-test-slave-2gms krb5kdc[12236](info): commencing operation
krb5kdc: starting...
W20260619 03:01:39.882963 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:40 dist-test-slave-2gms krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:40.115556 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:40.419282 12249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:40.419574 12249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:40.419736 12249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:40.426357 12249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:40.426496 12249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:40.426640 12249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:40.426769 12249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:40.426854 12249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:40.435781 12249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:40.436877 12249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:40.438060 12249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:40.449143 12257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:40.449154 12255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:40.449657 12254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:40.450680 12249 server_base.cc:1034] running on GCE node
I20260619 03:01:40.451373 12249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:40.452641 12249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:40.453863 12249 hybrid_clock.cc:648] HybridClock initialized: now 1781838100453826 us; error 51 us; skew 500 ppm
I20260619 03:01:40.454197 12249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:40 dist-test-slave-2gms krb5kdc[12236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:40.457255 12249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:40.458626 12249 webserver.cc:460] Webserver started at http://127.0.0.1:36831/ using document root <none> and password file <none>
I20260619 03:01:40.459276 12249 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:40.459437 12249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:40.459800 12249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:40.462944 12249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "06048f7755de4a26b6a21b19e9e8fec8"
format_stamp: "Formatted at 2026-06-19 03:01:40 on dist-test-slave-2gms"
server_key: "048705ee191152f14e00babba8bf608b"
server_key_iv: "26a623fa2518c0884aa584999ca90f72"
server_key_version: "encryptionkey@0"
I20260619 03:01:40.463698 12249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "06048f7755de4a26b6a21b19e9e8fec8"
format_stamp: "Formatted at 2026-06-19 03:01:40 on dist-test-slave-2gms"
server_key: "048705ee191152f14e00babba8bf608b"
server_key_iv: "26a623fa2518c0884aa584999ca90f72"
server_key_version: "encryptionkey@0"
I20260619 03:01:40.468626 12249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 03:01:40.471771 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:40.472891 12249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:40.473065 12249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "06048f7755de4a26b6a21b19e9e8fec8"
format_stamp: "Formatted at 2026-06-19 03:01:40 on dist-test-slave-2gms"
server_key: "048705ee191152f14e00babba8bf608b"
server_key_iv: "26a623fa2518c0884aa584999ca90f72"
server_key_version: "encryptionkey@0"
I20260619 03:01:40.473253 12249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:40.478471 12249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:40.481930 12249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:40.482194 12249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:40.499583 12249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32977
I20260619 03:01:40.499653 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32977 every 8 connection(s)
I20260619 03:01:40.500789 12249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:40.504258 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:40.504890 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12249
I20260619 03:01:40.505074 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:40.505484 23448 external_mini_cluster.cc:1428] Setting key 2ead2fc4333b78db642a909182954aa1
I20260619 03:01:40.512871 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:40.507119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:47937 (local address 127.0.0.1:32977)
0619 03:01:40.507866 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:40.507876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:40.507915 (+ 39us) server_negotiation.cc:407] Connection header received
0619 03:01:40.509893 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:40.509915 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:40.510212 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:40.510561 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:40.511470 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:40.511492 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:40.511505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:40.511518 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:40.511567 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:40.511588 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:40.511805 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:40.511929 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 03:01:40.512090 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":305,"threads_started":1}
I20260619 03:01:40.516006 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8: Bootstrap starting.
I20260619 03:01:40.516175 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:40.518895 12316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:40.519819 12316 log.cc:824] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:40.522374 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8: No bootstrap required, opened a new log
I20260619 03:01:40.522754 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:40.520321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:49511 (local address 127.0.0.1:32977)
0619 03:01:40.520526 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:40.520534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:40.521038 (+ 504us) server_negotiation.cc:407] Connection header received
0619 03:01:40.521205 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:40.521209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:40.521269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:40.521359 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:40.522166 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:40.522171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:40.522175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:40.522191 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:40.522232 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:40.522251 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:40.522360 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:40.522444 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 03:01:40.522495 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 03:01:40.529656 12316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06048f7755de4a26b6a21b19e9e8fec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260619 03:01:40.529994 12316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:40.530094 12316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06048f7755de4a26b6a21b19e9e8fec8, State: Initialized, Role: FOLLOWER
I20260619 03:01:40.530618 12316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06048f7755de4a26b6a21b19e9e8fec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260619 03:01:40.530864 12316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:40.530997 12316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:40.531165 12316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:40.533138 12316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06048f7755de4a26b6a21b19e9e8fec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260619 03:01:40.533676 12316 leader_election.cc:302] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06048f7755de4a26b6a21b19e9e8fec8; no voters:
I20260619 03:01:40.534229 12316 leader_election.cc:288] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:40.534325 12326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:40.535040 12326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [term 1 LEADER]: Becoming Leader. State: Replica: 06048f7755de4a26b6a21b19e9e8fec8, State: Running, Role: LEADER
I20260619 03:01:40.535553 12326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06048f7755de4a26b6a21b19e9e8fec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } }
I20260619 03:01:40.536115 12316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:40.538117 12328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06048f7755de4a26b6a21b19e9e8fec8. Latest consensus state: current_term: 1 leader_uuid: "06048f7755de4a26b6a21b19e9e8fec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06048f7755de4a26b6a21b19e9e8fec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } } }
I20260619 03:01:40.538200 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06048f7755de4a26b6a21b19e9e8fec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06048f7755de4a26b6a21b19e9e8fec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32977 } } }
I20260619 03:01:40.538535 12328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:40.538540 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:40.545424 12333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:40.551026 12333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:40.559870 12333 catalog_manager.cc:1347] Generated new cluster ID: 1ab92aaf72784164aee4255622cb5a43
I20260619 03:01:40.560057 12333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:40.572058 12333 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:40.572460 12333 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:40.581734 12333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06048f7755de4a26b6a21b19e9e8fec8: Generated new TSK 0
I20260619 03:01:40.582481 12333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:40.590646 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12249
W20260619 03:01:40.591702 12311 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.210:47937
W20260619 03:01:40.592353 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:40.592484 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12249
2026-06-19T03:01:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1775 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:40 dist-test-slave-2gms krb5kdc[12347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:40 dist-test-slave-2gms krb5kdc[12347](info): set up 2 sockets
Jun 19 03:01:40 dist-test-slave-2gms krb5kdc[12347](info): commencing operation
krb5kdc: starting...
W20260619 03:01:41.654727 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:41 dist-test-slave-2gms krb5kdc[12347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:41.885288 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:42.207412 12360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:42.207715 12360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:42.207877 12360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:42.215497 12360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:42.215626 12360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:42.215706 12360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:42.215798 12360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:42.215883 12360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:42.225657 12360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:42.226917 12360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:42.228147 12360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:42.239454 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:42.239475 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:42.239470 12368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:42.240294 12360 server_base.cc:1034] running on GCE node
I20260619 03:01:42.241003 12360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:42.242249 12360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:42.243461 12360 hybrid_clock.cc:648] HybridClock initialized: now 1781838102243430 us; error 44 us; skew 500 ppm
I20260619 03:01:42.243798 12360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:42 dist-test-slave-2gms krb5kdc[12347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:42.250624 12360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:42.251983 12360 webserver.cc:460] Webserver started at http://127.0.0.1:44849/ using document root <none> and password file <none>
I20260619 03:01:42.252589 12360 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:42.252728 12360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:42.253005 12360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:42.255813 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "a39dc5edfff8409aa737cb947f883bfa"
format_stamp: "Formatted at 2026-06-19 03:01:42 on dist-test-slave-2gms"
server_key: "966bf46329be96f6cb62699cee27b99d"
server_key_iv: "e9a107ae15b4a87481895e93d5be506d"
server_key_version: "encryptionkey@0"
I20260619 03:01:42.256603 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "a39dc5edfff8409aa737cb947f883bfa"
format_stamp: "Formatted at 2026-06-19 03:01:42 on dist-test-slave-2gms"
server_key: "966bf46329be96f6cb62699cee27b99d"
server_key_iv: "e9a107ae15b4a87481895e93d5be506d"
server_key_version: "encryptionkey@0"
I20260619 03:01:42.261368 12360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 03:01:42.264503 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:42.265703 12360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:42.265868 12360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "a39dc5edfff8409aa737cb947f883bfa"
format_stamp: "Formatted at 2026-06-19 03:01:42 on dist-test-slave-2gms"
server_key: "966bf46329be96f6cb62699cee27b99d"
server_key_iv: "e9a107ae15b4a87481895e93d5be506d"
server_key_version: "encryptionkey@0"
I20260619 03:01:42.266063 12360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:42.280320 12360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:42.283735 12360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:42.283975 12360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:42.300837 12360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35031
I20260619 03:01:42.300879 12426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35031 every 8 connection(s)
I20260619 03:01:42.302028 12360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:42.305150 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12360
I20260619 03:01:42.305318 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:42.305546 12427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:42.305869 23448 external_mini_cluster.cc:1428] Setting key bc41de490394bcdce14843b6c40d93b7
Jun 19 03:01:42 dist-test-slave-2gms krb5kdc[12347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 03:01:42.319449 12427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa: Bootstrap starting.
I20260619 03:01:42.323220 12427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:42.324494 12427 log.cc:824] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:42.327080 12427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa: No bootstrap required, opened a new log
I20260619 03:01:42.328186 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:42.307504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:41743 (local address 127.0.0.1:35031)
0619 03:01:42.308290 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:42.308300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:42.308342 (+ 42us) server_negotiation.cc:407] Connection header received
0619 03:01:42.310048 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:42.310067 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:42.310428 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:42.310801 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:42.312465 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:42.313578 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:42.315232 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:42.315823 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:42.315938 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:42.320444 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:42.320454 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:42.320470 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:42.320511 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:42.324294 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:42.325060 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:42.325065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:42.325070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:42.325235 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:42.325666 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:42.325669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:42.325671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:42.326145 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:42.326459 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:42.326987 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 03:01:42.327311 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":385,"threads_started":1}
I20260619 03:01:42.331734 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:42.334870 12427 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39dc5edfff8409aa737cb947f883bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35031 } }
I20260619 03:01:42.335194 12427 raft_consensus.cc:374] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:42.335294 12427 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39dc5edfff8409aa737cb947f883bfa, State: Initialized, Role: FOLLOWER
I20260619 03:01:42.335939 12427 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39dc5edfff8409aa737cb947f883bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35031 } }
I20260619 03:01:42.336261 12427 raft_consensus.cc:388] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:42.336409 12427 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:42.336586 12427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:42.339195 12427 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39dc5edfff8409aa737cb947f883bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35031 } }
I20260619 03:01:42.339689 12427 leader_election.cc:302] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a39dc5edfff8409aa737cb947f883bfa; no voters:
I20260619 03:01:42.340233 12427 leader_election.cc:288] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:42.340353 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:42.341629 12437 raft_consensus.cc:686] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [term 1 LEADER]: Becoming Leader. State: Replica: a39dc5edfff8409aa737cb947f883bfa, State: Running, Role: LEADER
I20260619 03:01:42.342522 12427 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:42.342986 12437 consensus_queue.cc:237] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39dc5edfff8409aa737cb947f883bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35031 } }
I20260619 03:01:42.346340 12439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39dc5edfff8409aa737cb947f883bfa. Latest consensus state: current_term: 1 leader_uuid: "a39dc5edfff8409aa737cb947f883bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39dc5edfff8409aa737cb947f883bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35031 } } }
I20260619 03:01:42.346745 12439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:42.347360 12438 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a39dc5edfff8409aa737cb947f883bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39dc5edfff8409aa737cb947f883bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35031 } } }
I20260619 03:01:42.351145 12438 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:42.359820 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:42.335950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:37435 (local address 127.0.0.1:35031)
0619 03:01:42.336128 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:42.336134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:42.336565 (+ 431us) server_negotiation.cc:407] Connection header received
0619 03:01:42.336770 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:42.336775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:42.336847 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:42.336961 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:42.338241 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:42.339349 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:42.340916 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:42.341896 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:42.342020 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:42.344542 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:42.344550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:42.344558 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:42.344603 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:42.354291 (+ 9688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:42.355503 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:42.355510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:42.355516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:42.355679 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:42.356390 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:42.356398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:42.356403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:42.356941 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:42.358617 (+ 1676us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:42.358944 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 03:01:42.359164 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 03:01:42.360584 12444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:42.366874 12444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:42.375890 12444 catalog_manager.cc:1347] Generated new cluster ID: 1efa8dd55b19415bb7c1692ae52b5c10
I20260619 03:01:42.376045 12444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:42.387306 12444 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:42.388417 12444 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:42.393967 12444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a39dc5edfff8409aa737cb947f883bfa: Generated new TSK 0
I20260619 03:01:42.394861 12444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:42.404366 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12360
I20260619 03:01:42.405480 12422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:42.491411 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12360
W20260619 03:01:42.508777 12345 connection.cc:535] client connection to 127.0.0.1:35031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35031): Connection reset by peer (error 104)
2026-06-19T03:01:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:42 dist-test-slave-2gms krb5kdc[12459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:42 dist-test-slave-2gms krb5kdc[12459](info): set up 2 sockets
Jun 19 03:01:42 dist-test-slave-2gms krb5kdc[12459](info): commencing operation
krb5kdc: starting...
W20260619 03:01:43.554935 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:43 dist-test-slave-2gms krb5kdc[12459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:43.786840 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:44.098311 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:44.098625 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:44.098802 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:44.105342 12472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:44.105471 12472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:44.105571 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:44.105672 12472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:44.105754 12472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:44.114710 12472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:44.115840 12472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:44.117026 12472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:44.128068 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:44.128075 12478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:44.128461 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:44.128649 12472 server_base.cc:1034] running on GCE node
I20260619 03:01:44.129340 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:44.130659 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:44.131871 12472 hybrid_clock.cc:648] HybridClock initialized: now 1781838104131832 us; error 47 us; skew 500 ppm
I20260619 03:01:44.132192 12472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:44 dist-test-slave-2gms krb5kdc[12459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:44.134943 12472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:44.136188 12472 webserver.cc:460] Webserver started at http://127.0.0.1:37303/ using document root <none> and password file <none>
I20260619 03:01:44.136802 12472 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:44.136940 12472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:44.137210 12472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:44.139982 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "e22da9727bb34655b083b41c4294fb2d"
format_stamp: "Formatted at 2026-06-19 03:01:44 on dist-test-slave-2gms"
server_key: "ab6c08b35622fc95a55f567e773539f1"
server_key_iv: "3f36ca5f9d76c3cb757852bb8af962b5"
server_key_version: "encryptionkey@0"
I20260619 03:01:44.140671 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "e22da9727bb34655b083b41c4294fb2d"
format_stamp: "Formatted at 2026-06-19 03:01:44 on dist-test-slave-2gms"
server_key: "ab6c08b35622fc95a55f567e773539f1"
server_key_iv: "3f36ca5f9d76c3cb757852bb8af962b5"
server_key_version: "encryptionkey@0"
I20260619 03:01:44.145648 12472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 03:01:44.148795 12487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:44.149958 12472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 03:01:44.150128 12472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "e22da9727bb34655b083b41c4294fb2d"
format_stamp: "Formatted at 2026-06-19 03:01:44 on dist-test-slave-2gms"
server_key: "ab6c08b35622fc95a55f567e773539f1"
server_key_iv: "3f36ca5f9d76c3cb757852bb8af962b5"
server_key_version: "encryptionkey@0"
I20260619 03:01:44.150322 12472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:44.167097 12472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:44.170699 12472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:44.170981 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:44.188328 12472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35049
I20260619 03:01:44.188350 12538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35049 every 8 connection(s)
I20260619 03:01:44.189548 12472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:44.192839 12539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:44.197350 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12472
I20260619 03:01:44.197546 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:44.198084 23448 external_mini_cluster.cc:1428] Setting key 814622997c08d6bf8f757c545d1f13db
I20260619 03:01:44.203213 12539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d: Bootstrap starting.
I20260619 03:01:44.206410 12539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:44.207525 12539 log.cc:824] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:44.210311 12539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d: No bootstrap required, opened a new log
I20260619 03:01:44.210834 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:44.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:55835 (local address 127.0.0.1:35049)
0619 03:01:44.200478 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:44.200488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:44.200525 (+ 37us) server_negotiation.cc:407] Connection header received
0619 03:01:44.202189 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:44.202208 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:44.202548 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:44.202902 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:44.203438 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:44.204660 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:44.206555 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:44.207212 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:44.207352 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:44.208619 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:44.208630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:44.208652 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:44.209195 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:44.209256 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:44.209285 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:44.209390 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:44.209667 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 03:01:44.209832 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":339,"threads_started":1}
I20260619 03:01:44.214229 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:44.219455 12539 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22da9727bb34655b083b41c4294fb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260619 03:01:44.219925 12539 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:44.220049 12539 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22da9727bb34655b083b41c4294fb2d, State: Initialized, Role: FOLLOWER
I20260619 03:01:44.220578 12539 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22da9727bb34655b083b41c4294fb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260619 03:01:44.220835 12539 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:44.220978 12539 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:44.221175 12539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:44.223315 12539 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22da9727bb34655b083b41c4294fb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260619 03:01:44.223765 12539 leader_election.cc:302] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22da9727bb34655b083b41c4294fb2d; no voters:
I20260619 03:01:44.224395 12539 leader_election.cc:288] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:44.224483 12549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:44.225080 12549 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [term 1 LEADER]: Becoming Leader. State: Replica: e22da9727bb34655b083b41c4294fb2d, State: Running, Role: LEADER
I20260619 03:01:44.225831 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:44.219193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:43789 (local address 127.0.0.1:35049)
0619 03:01:44.219389 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:44.219395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:44.219738 (+ 343us) server_negotiation.cc:407] Connection header received
0619 03:01:44.219921 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:44.219926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:44.220011 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:44.220109 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:44.220675 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:44.221644 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:44.223032 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:44.223522 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:44.223633 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:44.224784 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:44.224788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:44.224791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:44.224810 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:44.224842 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:44.224876 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:44.225005 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:44.225498 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 03:01:44.225542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 03:01:44.225804 12549 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22da9727bb34655b083b41c4294fb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260619 03:01:44.227931 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22da9727bb34655b083b41c4294fb2d. Latest consensus state: current_term: 1 leader_uuid: "e22da9727bb34655b083b41c4294fb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22da9727bb34655b083b41c4294fb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } } }
I20260619 03:01:44.228322 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:44.228703 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22da9727bb34655b083b41c4294fb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22da9727bb34655b083b41c4294fb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } } }
I20260619 03:01:44.229075 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:44.232810 12539 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:44.235215 12553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:44.241040 12553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:44.248919 12553 catalog_manager.cc:1347] Generated new cluster ID: df8cc5db7fbb456bb177ae4ea92ec1a7
I20260619 03:01:44.249020 12553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:44.266422 12553 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:44.267591 12553 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:44.273716 12553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22da9727bb34655b083b41c4294fb2d: Generated new TSK 0
I20260619 03:01:44.274350 12553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:44.288462 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12472
W20260619 03:01:44.289572 12534 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.210:55835
W20260619 03:01:44.290411 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12472: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:44.290555 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12472
W20260619 03:01:44.307638 12457 connection.cc:535] client connection to 127.0.0.1:35049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35049): Connection reset by peer (error 104)
2026-06-19T03:01:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1798 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:44 dist-test-slave-2gms krb5kdc[12570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:44 dist-test-slave-2gms krb5kdc[12570](info): set up 2 sockets
Jun 19 03:01:44 dist-test-slave-2gms krb5kdc[12570](info): commencing operation
krb5kdc: starting...
W20260619 03:01:45.350972 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:45 dist-test-slave-2gms krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 03:01:45.575062 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:45.882380 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:45.882666 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:45.882822 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:45.889554 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:45.889674 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:45.889775 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:45.889887 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:45.889971 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:45.899163 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:45.900322 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:45.901579 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:45.913074 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:45.913120 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:45.913229 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:45.914233 12583 server_base.cc:1034] running on GCE node
I20260619 03:01:45.914952 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:45.916139 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:45.917351 12583 hybrid_clock.cc:648] HybridClock initialized: now 1781838105917289 us; error 49 us; skew 500 ppm
I20260619 03:01:45.917637 12583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:45 dist-test-slave-2gms krb5kdc[12570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:45.920650 12583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 03:01:45.922060 12583 webserver.cc:460] Webserver started at http://127.0.0.1:41077/ using document root <none> and password file <none>
I20260619 03:01:45.922767 12583 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:45.922901 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:45.923213 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:45.926609 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "1c96b77c14974832b6bc7de8dd334058"
format_stamp: "Formatted at 2026-06-19 03:01:45 on dist-test-slave-2gms"
server_key: "f50e32dd8361fa95674ba9c3e20e2998"
server_key_iv: "c3eb480abd8a8829c25696dd119b2378"
server_key_version: "encryptionkey@0"
I20260619 03:01:45.927394 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "1c96b77c14974832b6bc7de8dd334058"
format_stamp: "Formatted at 2026-06-19 03:01:45 on dist-test-slave-2gms"
server_key: "f50e32dd8361fa95674ba9c3e20e2998"
server_key_iv: "c3eb480abd8a8829c25696dd119b2378"
server_key_version: "encryptionkey@0"
I20260619 03:01:45.932175 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 03:01:45.935377 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:45.936553 12583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:45.936726 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "1c96b77c14974832b6bc7de8dd334058"
format_stamp: "Formatted at 2026-06-19 03:01:45 on dist-test-slave-2gms"
server_key: "f50e32dd8361fa95674ba9c3e20e2998"
server_key_iv: "c3eb480abd8a8829c25696dd119b2378"
server_key_version: "encryptionkey@0"
I20260619 03:01:45.936941 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:45.942193 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:45.945806 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:45.946077 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:45.963621 12583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33273
I20260619 03:01:45.963692 12649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33273 every 8 connection(s)
I20260619 03:01:45.964844 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:45.968308 12650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:45.969226 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12583
I20260619 03:01:45.969415 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:45.970021 23448 external_mini_cluster.cc:1428] Setting key df2418f7a94bd0bf4d6183e9c82403b2
I20260619 03:01:45.977226 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:45.971745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:48901 (local address 127.0.0.1:33273)
0619 03:01:45.972493 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:45.972503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:45.972541 (+ 38us) server_negotiation.cc:407] Connection header received
0619 03:01:45.974228 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:45.974249 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:45.974544 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:45.974926 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:45.975859 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:45.975864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:45.975875 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:45.975887 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:45.975946 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:45.975964 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:45.976108 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:45.976229 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 03:01:45.976407 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":335,"threads_started":1}
I20260619 03:01:45.980650 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:45.980711 12650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058: Bootstrap starting.
I20260619 03:01:45.984241 12650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:45.985530 12650 log.cc:824] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:45.987910 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:45.985590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.210:50111 (local address 127.0.0.1:33273)
0619 03:01:45.985728 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:45.985732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:45.986096 (+ 364us) server_negotiation.cc:407] Connection header received
0619 03:01:45.986281 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:45.986286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:45.986354 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:45.986454 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:45.987306 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:45.987310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:45.987313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:45.987325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:45.987366 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:45.987380 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:45.987461 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:45.987585 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 03:01:45.987621 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 03:01:45.988647 12650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058: No bootstrap required, opened a new log
I20260619 03:01:45.998024 12650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c96b77c14974832b6bc7de8dd334058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260619 03:01:45.998375 12650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:45.998481 12650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c96b77c14974832b6bc7de8dd334058, State: Initialized, Role: FOLLOWER
I20260619 03:01:45.999099 12650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c96b77c14974832b6bc7de8dd334058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260619 03:01:45.999344 12650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:45.999488 12650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:45.999655 12650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:46.001767 12650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c96b77c14974832b6bc7de8dd334058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260619 03:01:46.002208 12650 leader_election.cc:302] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c96b77c14974832b6bc7de8dd334058; no voters:
I20260619 03:01:46.002775 12650 leader_election.cc:288] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:46.002894 12660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:46.003535 12660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [term 1 LEADER]: Becoming Leader. State: Replica: 1c96b77c14974832b6bc7de8dd334058, State: Running, Role: LEADER
I20260619 03:01:46.004006 12660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c96b77c14974832b6bc7de8dd334058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } }
I20260619 03:01:46.004709 12650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:46.006829 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c96b77c14974832b6bc7de8dd334058. Latest consensus state: current_term: 1 leader_uuid: "1c96b77c14974832b6bc7de8dd334058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c96b77c14974832b6bc7de8dd334058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } } }
I20260619 03:01:46.006729 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c96b77c14974832b6bc7de8dd334058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c96b77c14974832b6bc7de8dd334058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33273 } } }
I20260619 03:01:46.007174 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:46.009418 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:46.012513 12666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:46.017288 12666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:46.025333 12666 catalog_manager.cc:1347] Generated new cluster ID: 91eeb81d1b1149f0910e4a9009474bdb
I20260619 03:01:46.025435 12666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:46.047627 12666 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:46.048032 12666 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:46.070163 12666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c96b77c14974832b6bc7de8dd334058: Generated new TSK 0
I20260619 03:01:46.071030 12666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:46.084066 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12583
W20260619 03:01:46.084996 12645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.210:48901
W20260619 03:01:46.085654 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:46.085807 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12583
2026-06-19T03:01:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1796 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:46 dist-test-slave-2gms krb5kdc[12681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:46 dist-test-slave-2gms krb5kdc[12681](info): set up 2 sockets
Jun 19 03:01:46 dist-test-slave-2gms krb5kdc[12681](info): commencing operation
krb5kdc: starting...
W20260619 03:01:47.171065 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:47 dist-test-slave-2gms krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:47.413762 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:37749
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:37749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:47.720650 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:47.720893 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:47.720973 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:47.727445 12694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:47.727525 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:47.727638 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:47.727732 12694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:47.727809 12694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:47.736482 12694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:37749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:37749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:47.737555 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:47.738781 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:47.749681 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:47.749755 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:47.750119 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:47.750458 12694 server_base.cc:1034] running on GCE node
I20260619 03:01:47.751140 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:47.752270 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:47.753458 12694 hybrid_clock.cc:648] HybridClock initialized: now 1781838107753424 us; error 31 us; skew 500 ppm
I20260619 03:01:47.753723 12694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:47 dist-test-slave-2gms krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838107, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:47.759718 12694 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:47.761174 12694 webserver.cc:460] Webserver started at http://127.22.230.62:34173/ using document root <none> and password file <none>
I20260619 03:01:47.761816 12694 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:47.761937 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:47.762228 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:47.765005 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "96834583aa514f3b80dd85c96e2a42a4"
format_stamp: "Formatted at 2026-06-19 03:01:47 on dist-test-slave-2gms"
server_key: "37b71db4e7488d78ea359459ccafa3f9"
server_key_iv: "2d2eab4a953d05c0e9f5c5b58332a459"
server_key_version: "encryptionkey@0"
I20260619 03:01:47.765726 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "96834583aa514f3b80dd85c96e2a42a4"
format_stamp: "Formatted at 2026-06-19 03:01:47 on dist-test-slave-2gms"
server_key: "37b71db4e7488d78ea359459ccafa3f9"
server_key_iv: "2d2eab4a953d05c0e9f5c5b58332a459"
server_key_version: "encryptionkey@0"
I20260619 03:01:47.770551 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 03:01:47.773767 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:47.775117 12694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 03:01:47.775293 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "96834583aa514f3b80dd85c96e2a42a4"
format_stamp: "Formatted at 2026-06-19 03:01:47 on dist-test-slave-2gms"
server_key: "37b71db4e7488d78ea359459ccafa3f9"
server_key_iv: "2d2eab4a953d05c0e9f5c5b58332a459"
server_key_version: "encryptionkey@0"
I20260619 03:01:47.775460 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:47.792586 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:47.796147 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:47.796396 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:47.814116 12694 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:37749
I20260619 03:01:47.814146 12760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:37749 every 8 connection(s)
I20260619 03:01:47.815394 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:47.818976 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:47.824395 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12694
I20260619 03:01:47.824594 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:47.825105 23448 external_mini_cluster.cc:1428] Setting key 1d9d379ecd62a752c01fbe73e68589d3
I20260619 03:01:47.829300 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4: Bootstrap starting.
I20260619 03:01:47.832747 12761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:47.833986 12761 log.cc:824] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:47.836864 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4: No bootstrap required, opened a new log
Jun 19 03:01:47 dist-test-slave-2gms krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:47.844676 12761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96834583aa514f3b80dd85c96e2a42a4" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37749 } }
I20260619 03:01:47.845017 12761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:47.845147 12761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96834583aa514f3b80dd85c96e2a42a4, State: Initialized, Role: FOLLOWER
I20260619 03:01:47.845808 12761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96834583aa514f3b80dd85c96e2a42a4" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37749 } }
I20260619 03:01:47.846112 12761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:47.846274 12761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:47.846460 12761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:47.847185 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:47.826840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44531 (local address 127.22.230.62:37749)
0619 03:01:47.827654 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:47.827665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:47.827701 (+ 36us) server_negotiation.cc:407] Connection header received
0619 03:01:47.829322 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:47.829342 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:47.829730 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:47.830087 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:47.831782 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:47.833051 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:47.834826 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:47.835582 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:47.835702 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:47.840129 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:47.840136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:47.840148 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:47.840179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:47.843111 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:47.843877 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:47.843889 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:47.843894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:47.844042 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:47.844497 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:47.844500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:47.844502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:47.844935 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:47.845287 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:47.845747 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 03:01:47.846060 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":412,"threads_started":1}
I20260619 03:01:47.849203 12761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96834583aa514f3b80dd85c96e2a42a4" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37749 } }
I20260619 03:01:47.849761 12761 leader_election.cc:302] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96834583aa514f3b80dd85c96e2a42a4; no voters:
I20260619 03:01:47.850412 12761 leader_election.cc:288] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:47.850534 12766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:47.850943 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:47.851734 12766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [term 1 LEADER]: Becoming Leader. State: Replica: 96834583aa514f3b80dd85c96e2a42a4, State: Running, Role: LEADER
I20260619 03:01:47.852257 12766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96834583aa514f3b80dd85c96e2a42a4" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37749 } }
I20260619 03:01:47.852612 12761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:47.858716 12767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96834583aa514f3b80dd85c96e2a42a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96834583aa514f3b80dd85c96e2a42a4" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37749 } } }
I20260619 03:01:47.859400 12767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:47.858942 12768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96834583aa514f3b80dd85c96e2a42a4. Latest consensus state: current_term: 1 leader_uuid: "96834583aa514f3b80dd85c96e2a42a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96834583aa514f3b80dd85c96e2a42a4" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 37749 } } }
I20260619 03:01:47.859930 12768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:47.862855 12778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:47.869184 12778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:47.878749 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:47.860297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55955 (local address 127.22.230.62:37749)
0619 03:01:47.861449 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:47.861455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:47.862075 (+ 620us) server_negotiation.cc:407] Connection header received
0619 03:01:47.862282 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:47.862290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:47.862384 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:47.862491 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:47.864224 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:47.865100 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:47.866930 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:47.867617 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:47.867712 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:47.871070 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:47.871076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:47.871081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:47.871119 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:47.875016 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:47.875915 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:47.875921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:47.875924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:47.876045 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:47.876558 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:47.876564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:47.876568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:47.876945 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:47.877206 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:47.878233 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0619 03:01:47.878404 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052}
I20260619 03:01:47.880072 12788 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 03:01:47.882258 12778 catalog_manager.cc:1347] Generated new cluster ID: 18dd9044028b4ff3944a755c09a4e1d9
I20260619 03:01:47.882407 12778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:47.904714 12778 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:47.905812 12778 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:47.921538 12778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96834583aa514f3b80dd85c96e2a42a4: Generated new TSK 0
I20260619 03:01:47.922412 12778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:47.934471 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12694
I20260619 03:01:47.935647 12756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:48.024516 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12694
W20260619 03:01:48.039217 12679 connection.cc:535] client connection to 127.22.230.62:37749 recv error: Network error: failed to read from TLS socket (remote: 127.22.230.62:37749): Connection reset by peer (error 104)
2026-06-19T03:01:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:48 dist-test-slave-2gms krb5kdc[12793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:48 dist-test-slave-2gms krb5kdc[12793](info): set up 2 sockets
Jun 19 03:01:48 dist-test-slave-2gms krb5kdc[12793](info): commencing operation
krb5kdc: starting...
W20260619 03:01:49.094885 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:49 dist-test-slave-2gms krb5kdc[12793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:49.325486 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45537
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:49.630138 12806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:49.630362 12806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:49.630436 12806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:49.636796 12806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:49.636878 12806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:49.636917 12806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:49.636962 12806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:49.636994 12806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:49.645354 12806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:45537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:49.646353 12806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:49.647518 12806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:49.657998 12811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:49.658087 12812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:49.658052 12814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:49.658771 12806 server_base.cc:1034] running on GCE node
I20260619 03:01:49.659548 12806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:49.660651 12806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:49.661840 12806 hybrid_clock.cc:648] HybridClock initialized: now 1781838109661806 us; error 39 us; skew 500 ppm
I20260619 03:01:49.662134 12806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:49 dist-test-slave-2gms krb5kdc[12793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838109, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:49.664896 12806 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:49.666090 12806 webserver.cc:460] Webserver started at http://127.22.230.62:46429/ using document root <none> and password file <none>
I20260619 03:01:49.666730 12806 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:49.666826 12806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:49.667050 12806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:49.669621 12806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "4ef3e611f3c548d58cf8621f4000a90c"
format_stamp: "Formatted at 2026-06-19 03:01:49 on dist-test-slave-2gms"
server_key: "bfbb5f3af352bcf10597d9199c23ffd5"
server_key_iv: "a64dace1ee3122d435e396d9e80ab110"
server_key_version: "encryptionkey@0"
I20260619 03:01:49.670260 12806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "4ef3e611f3c548d58cf8621f4000a90c"
format_stamp: "Formatted at 2026-06-19 03:01:49 on dist-test-slave-2gms"
server_key: "bfbb5f3af352bcf10597d9199c23ffd5"
server_key_iv: "a64dace1ee3122d435e396d9e80ab110"
server_key_version: "encryptionkey@0"
I20260619 03:01:49.675084 12806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 03:01:49.678069 12821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:49.679318 12806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:49.679492 12806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "4ef3e611f3c548d58cf8621f4000a90c"
format_stamp: "Formatted at 2026-06-19 03:01:49 on dist-test-slave-2gms"
server_key: "bfbb5f3af352bcf10597d9199c23ffd5"
server_key_iv: "a64dace1ee3122d435e396d9e80ab110"
server_key_version: "encryptionkey@0"
I20260619 03:01:49.679672 12806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:49.707206 12806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:49.710700 12806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:49.710947 12806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:49.728297 12806 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:45537
I20260619 03:01:49.728358 12872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:45537 every 8 connection(s)
I20260619 03:01:49.729558 12806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:49.732873 12873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:49.735527 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12806
I20260619 03:01:49.735710 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:49.736193 23448 external_mini_cluster.cc:1428] Setting key 95917510d97896db2fbdf333b609d5ff
I20260619 03:01:49.743923 12873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c: Bootstrap starting.
I20260619 03:01:49.747215 12873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:49.748167 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:49.737761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49557 (local address 127.22.230.62:45537)
0619 03:01:49.738491 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:49.738501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:49.738539 (+ 38us) server_negotiation.cc:407] Connection header received
0619 03:01:49.740141 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:49.740161 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:49.740505 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:49.740794 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:49.741352 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:49.742653 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:49.744298 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:49.744898 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:49.745007 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:49.746156 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:49.746162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:49.746173 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:49.746677 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:49.746753 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:49.746779 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:49.746920 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:49.747194 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 03:01:49.747347 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":346,"threads_started":1}
I20260619 03:01:49.748171 12873 log.cc:824] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:49.751058 12873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c: No bootstrap required, opened a new log
I20260619 03:01:49.751631 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:49.758150 12873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3e611f3c548d58cf8621f4000a90c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45537 } }
I20260619 03:01:49.758500 12873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:49.758616 12873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef3e611f3c548d58cf8621f4000a90c, State: Initialized, Role: FOLLOWER
I20260619 03:01:49.759213 12873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3e611f3c548d58cf8621f4000a90c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45537 } }
I20260619 03:01:49.759481 12873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:49.759647 12873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:49.759824 12873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:49.761265 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:49.755537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57633 (local address 127.22.230.62:45537)
0619 03:01:49.755691 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:49.755697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:49.756068 (+ 371us) server_negotiation.cc:407] Connection header received
0619 03:01:49.756251 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:49.756254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:49.756306 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:49.756411 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:49.756843 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:49.757523 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:49.758973 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:49.759402 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:49.759558 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:49.760579 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:49.760583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:49.760586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:49.760600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:49.760646 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:49.760659 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:49.760739 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:49.760917 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 03:01:49.760950 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 03:01:49.762538 12873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3e611f3c548d58cf8621f4000a90c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45537 } }
I20260619 03:01:49.763058 12873 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef3e611f3c548d58cf8621f4000a90c; no voters:
I20260619 03:01:49.763537 12873 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:49.763621 12883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:49.764207 12883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [term 1 LEADER]: Becoming Leader. State: Replica: 4ef3e611f3c548d58cf8621f4000a90c, State: Running, Role: LEADER
I20260619 03:01:49.764744 12883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3e611f3c548d58cf8621f4000a90c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45537 } }
I20260619 03:01:49.766582 12873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:49.766402 12884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef3e611f3c548d58cf8621f4000a90c. Latest consensus state: current_term: 1 leader_uuid: "4ef3e611f3c548d58cf8621f4000a90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3e611f3c548d58cf8621f4000a90c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45537 } } }
I20260619 03:01:49.766922 12884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:49.767387 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef3e611f3c548d58cf8621f4000a90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef3e611f3c548d58cf8621f4000a90c" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 45537 } } }
I20260619 03:01:49.767675 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:49.773799 12887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:49.778653 12887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:49.786113 12887 catalog_manager.cc:1347] Generated new cluster ID: c0da565f16e74882bd9b7ea0524c1b15
I20260619 03:01:49.786216 12887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:49.799127 12887 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:49.800767 12887 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:49.807260 12887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef3e611f3c548d58cf8621f4000a90c: Generated new TSK 0
I20260619 03:01:49.808122 12887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:49.813062 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12806
W20260619 03:01:49.814103 12868 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49557
W20260619 03:01:49.814935 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12806: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:49.815125 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12806
W20260619 03:01:49.829277 12791 connection.cc:535] client connection to 127.22.230.62:45537 recv error: Network error: failed to read from TLS socket (remote: 127.22.230.62:45537): Connection reset by peer (error 104)
2026-06-19T03:01:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1789 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:49 dist-test-slave-2gms krb5kdc[12904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:49 dist-test-slave-2gms krb5kdc[12904](info): set up 2 sockets
Jun 19 03:01:49 dist-test-slave-2gms krb5kdc[12904](info): commencing operation
krb5kdc: starting...
W20260619 03:01:50.887040 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:51 dist-test-slave-2gms krb5kdc[12904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:51.121857 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38037
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:38037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:51.426345 12917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:51.426648 12917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:51.426769 12917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:51.433241 12917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:51.433320 12917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:51.433360 12917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:51.433420 12917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:51.433458 12917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:51.442121 12917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:38037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:38037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:51.443151 12917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:51.444245 12917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:51.455240 12922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:51.455157 12925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:51.455210 12923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:51.455947 12917 server_base.cc:1034] running on GCE node
I20260619 03:01:51.456643 12917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:51.457906 12917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:51.459141 12917 hybrid_clock.cc:648] HybridClock initialized: now 1781838111459088 us; error 42 us; skew 500 ppm
I20260619 03:01:51.459522 12917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:51 dist-test-slave-2gms krb5kdc[12904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838111, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:51.462502 12917 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:51.463824 12917 webserver.cc:460] Webserver started at http://127.22.230.62:35653/ using document root <none> and password file <none>
I20260619 03:01:51.464448 12917 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:51.464588 12917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:51.464888 12917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:51.467679 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "5ced182a2b9e4be18533c6517eb1368f"
format_stamp: "Formatted at 2026-06-19 03:01:51 on dist-test-slave-2gms"
server_key: "399b4f6885a9d5d4aae8fbd7b866de0b"
server_key_iv: "68159003640fe9c069a67f93005d92a6"
server_key_version: "encryptionkey@0"
I20260619 03:01:51.468425 12917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "5ced182a2b9e4be18533c6517eb1368f"
format_stamp: "Formatted at 2026-06-19 03:01:51 on dist-test-slave-2gms"
server_key: "399b4f6885a9d5d4aae8fbd7b866de0b"
server_key_iv: "68159003640fe9c069a67f93005d92a6"
server_key_version: "encryptionkey@0"
I20260619 03:01:51.473220 12917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 03:01:51.476282 12932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:51.477519 12917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:01:51.477674 12917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "5ced182a2b9e4be18533c6517eb1368f"
format_stamp: "Formatted at 2026-06-19 03:01:51 on dist-test-slave-2gms"
server_key: "399b4f6885a9d5d4aae8fbd7b866de0b"
server_key_iv: "68159003640fe9c069a67f93005d92a6"
server_key_version: "encryptionkey@0"
I20260619 03:01:51.477892 12917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:51.483026 12917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:51.486867 12917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:51.487146 12917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:51.504503 12917 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:38037
I20260619 03:01:51.504539 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:38037 every 8 connection(s)
I20260619 03:01:51.505673 12917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:51.508824 12984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:51.510375 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 12917
I20260619 03:01:51.510603 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:51.511016 23448 external_mini_cluster.cc:1428] Setting key 13b16542af83fffe80c2d1fd924cf421
I20260619 03:01:51.517882 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:51.512575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43023 (local address 127.22.230.62:38037)
0619 03:01:51.513311 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:51.513321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:51.513356 (+ 35us) server_negotiation.cc:407] Connection header received
0619 03:01:51.514942 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:51.514961 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:51.515269 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:51.515562 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:51.516474 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:51.516480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:51.516491 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:51.516503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:51.516556 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:51.516573 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:51.516723 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:51.516855 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 03:01:51.517022 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":355,"threads_started":1}
I20260619 03:01:51.521067 12984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f: Bootstrap starting.
I20260619 03:01:51.521258 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:51.523939 12984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:51.524842 12984 log.cc:824] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:51.527629 12984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f: No bootstrap required, opened a new log
I20260619 03:01:51.527948 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:51.525472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51639 (local address 127.22.230.62:38037)
0619 03:01:51.525626 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:51.525631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:51.526113 (+ 482us) server_negotiation.cc:407] Connection header received
0619 03:01:51.526288 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:51.526299 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:51.526338 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:51.526450 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:51.527304 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:51.527309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:51.527313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:51.527329 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:51.527372 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:51.527388 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:51.527499 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:51.527587 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 03:01:51.527643 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 03:01:51.534394 12984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ced182a2b9e4be18533c6517eb1368f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38037 } }
I20260619 03:01:51.534754 12984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:51.534854 12984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ced182a2b9e4be18533c6517eb1368f, State: Initialized, Role: FOLLOWER
I20260619 03:01:51.535451 12984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ced182a2b9e4be18533c6517eb1368f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38037 } }
I20260619 03:01:51.535719 12984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:51.535868 12984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:51.536056 12984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:51.537983 12984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ced182a2b9e4be18533c6517eb1368f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38037 } }
I20260619 03:01:51.538430 12984 leader_election.cc:302] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ced182a2b9e4be18533c6517eb1368f; no voters:
I20260619 03:01:51.539011 12984 leader_election.cc:288] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:51.539149 12994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:51.540138 12994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [term 1 LEADER]: Becoming Leader. State: Replica: 5ced182a2b9e4be18533c6517eb1368f, State: Running, Role: LEADER
I20260619 03:01:51.540596 12994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ced182a2b9e4be18533c6517eb1368f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38037 } }
I20260619 03:01:51.540737 12984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:51.543136 12996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ced182a2b9e4be18533c6517eb1368f. Latest consensus state: current_term: 1 leader_uuid: "5ced182a2b9e4be18533c6517eb1368f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ced182a2b9e4be18533c6517eb1368f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38037 } } }
I20260619 03:01:51.543502 12996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:51.543978 12995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ced182a2b9e4be18533c6517eb1368f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ced182a2b9e4be18533c6517eb1368f" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 38037 } } }
I20260619 03:01:51.544263 12995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:51.544521 13002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:51.550873 13002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:51.558764 13002 catalog_manager.cc:1347] Generated new cluster ID: ef7b67e159944c2cb2f4bdeacdfcd899
I20260619 03:01:51.558862 13002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:51.569082 13002 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:51.569453 13002 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:51.582438 13002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ced182a2b9e4be18533c6517eb1368f: Generated new TSK 0
I20260619 03:01:51.583177 13002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:51.586994 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12917
W20260619 03:01:51.588452 12979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43023
W20260619 03:01:51.589085 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 12917: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:51.589214 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 12917
2026-06-19T03:01:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:51 dist-test-slave-2gms krb5kdc[13015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:51 dist-test-slave-2gms krb5kdc[13015](info): set up 2 sockets
Jun 19 03:01:51 dist-test-slave-2gms krb5kdc[13015](info): commencing operation
krb5kdc: starting...
W20260619 03:01:52.658931 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:52 dist-test-slave-2gms krb5kdc[13015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:52.889449 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33609
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:33609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:53.194424 13028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:53.194716 13028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:53.194878 13028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:53.201342 13028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:53.201435 13028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:53.201480 13028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:53.201563 13028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:53.201627 13028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:53.210464 13028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:33609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:33609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:53.211498 13028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:53.212556 13028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:53.223301 13033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:53.223351 13034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:53.223372 13036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:53.223989 13028 server_base.cc:1034] running on GCE node
I20260619 03:01:53.224607 13028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:53.225854 13028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:53.227069 13028 hybrid_clock.cc:648] HybridClock initialized: now 1781838113227039 us; error 41 us; skew 500 ppm
I20260619 03:01:53.227386 13028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:53 dist-test-slave-2gms krb5kdc[13015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838113, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:53.233084 13028 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:53.234335 13028 webserver.cc:460] Webserver started at http://127.22.230.62:43641/ using document root <none> and password file <none>
I20260619 03:01:53.234998 13028 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:53.235140 13028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:53.235415 13028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:53.238142 13028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "a0e71f4d71c54b748243a62ed31f9848"
format_stamp: "Formatted at 2026-06-19 03:01:53 on dist-test-slave-2gms"
server_key: "9bcbdf49954d1c6b25c0cff8007841fa"
server_key_iv: "225788babbbdd64cbf681928212f6473"
server_key_version: "encryptionkey@0"
I20260619 03:01:53.238940 13028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "a0e71f4d71c54b748243a62ed31f9848"
format_stamp: "Formatted at 2026-06-19 03:01:53 on dist-test-slave-2gms"
server_key: "9bcbdf49954d1c6b25c0cff8007841fa"
server_key_iv: "225788babbbdd64cbf681928212f6473"
server_key_version: "encryptionkey@0"
I20260619 03:01:53.243745 13028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 03:01:53.246941 13043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:53.248090 13028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 03:01:53.248263 13028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "a0e71f4d71c54b748243a62ed31f9848"
format_stamp: "Formatted at 2026-06-19 03:01:53 on dist-test-slave-2gms"
server_key: "9bcbdf49954d1c6b25c0cff8007841fa"
server_key_iv: "225788babbbdd64cbf681928212f6473"
server_key_version: "encryptionkey@0"
I20260619 03:01:53.248450 13028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:53.302608 13028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:53.306318 13028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:53.306598 13028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:53.324530 13028 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:33609
I20260619 03:01:53.324563 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:33609 every 8 connection(s)
I20260619 03:01:53.325748 13028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:53.329131 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:53.331483 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13028
I20260619 03:01:53.331687 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:53.332099 23448 external_mini_cluster.cc:1428] Setting key b1e1f563bf6736410feae5d22a526bd0
I20260619 03:01:53.340147 13095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848: Bootstrap starting.
I20260619 03:01:53.343617 13095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:53.344820 13095 log.cc:824] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848: Log is configured to *not* fsync() on all Append() calls
Jun 19 03:01:53 dist-test-slave-2gms krb5kdc[13015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:53.347554 13095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848: No bootstrap required, opened a new log
I20260619 03:01:53.354449 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:53.333808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35299 (local address 127.22.230.62:33609)
0619 03:01:53.334653 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:53.334663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:53.334700 (+ 37us) server_negotiation.cc:407] Connection header received
0619 03:01:53.336235 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:53.336253 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:53.336592 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:53.336868 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:53.338529 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:53.339842 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:53.341593 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:53.342234 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:53.342362 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:53.347117 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:53.347124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:53.347136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:53.347170 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:53.350132 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:53.350829 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:53.350845 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:53.350850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:53.350999 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:53.351461 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:53.351466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:53.351470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:53.352148 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:53.352465 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:53.352889 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 03:01:53.353268 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":378,"threads_started":1}
I20260619 03:01:53.356200 13095 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e71f4d71c54b748243a62ed31f9848" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33609 } }
I20260619 03:01:53.356573 13095 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:53.356665 13095 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e71f4d71c54b748243a62ed31f9848, State: Initialized, Role: FOLLOWER
I20260619 03:01:53.357254 13095 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e71f4d71c54b748243a62ed31f9848" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33609 } }
I20260619 03:01:53.357486 13095 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:53.357637 13095 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:53.357813 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:53.357798 13095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:53.360773 13095 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e71f4d71c54b748243a62ed31f9848" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33609 } }
I20260619 03:01:53.361363 13095 leader_election.cc:302] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e71f4d71c54b748243a62ed31f9848; no voters:
I20260619 03:01:53.362092 13095 leader_election.cc:288] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:53.362202 13104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:53.364043 13104 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [term 1 LEADER]: Becoming Leader. State: Replica: a0e71f4d71c54b748243a62ed31f9848, State: Running, Role: LEADER
I20260619 03:01:53.364434 13095 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:53.364634 13104 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e71f4d71c54b748243a62ed31f9848" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33609 } }
I20260619 03:01:53.380107 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:53.363215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44657 (local address 127.22.230.62:33609)
0619 03:01:53.363421 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:53.363429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:53.364011 (+ 582us) server_negotiation.cc:407] Connection header received
0619 03:01:53.364257 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:53.364262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:53.364353 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:53.364429 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:53.366643 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:53.368113 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:53.370340 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:53.371214 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:53.371348 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:53.374174 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:53.374180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:53.374184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:53.374214 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:53.376953 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:53.377571 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:53.377575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:53.377578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:53.377696 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:53.378084 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:53.378089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:53.378092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:53.378382 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:53.378555 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:53.379123 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 03:01:53.379358 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:01:53.371989 13106 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e71f4d71c54b748243a62ed31f9848. Latest consensus state: current_term: 1 leader_uuid: "a0e71f4d71c54b748243a62ed31f9848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e71f4d71c54b748243a62ed31f9848" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33609 } } }
I20260619 03:01:53.381968 13106 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:53.379630 13105 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0e71f4d71c54b748243a62ed31f9848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e71f4d71c54b748243a62ed31f9848" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 33609 } } }
I20260619 03:01:53.382694 13105 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:53.383250 13118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:53.387240 13118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:53.394311 13118 catalog_manager.cc:1347] Generated new cluster ID: 09ee2698acf54206b65af175a26a153f
I20260619 03:01:53.394400 13118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:53.415966 13118 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:53.417111 13118 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:53.423588 13118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0e71f4d71c54b748243a62ed31f9848: Generated new TSK 0
I20260619 03:01:53.424228 13118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:53.438877 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13028
I20260619 03:01:53.440117 13090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:01:53.526859 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13028
2026-06-19T03:01:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:53 dist-test-slave-2gms krb5kdc[13127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:53 dist-test-slave-2gms krb5kdc[13127](info): set up 2 sockets
Jun 19 03:01:53 dist-test-slave-2gms krb5kdc[13127](info): commencing operation
krb5kdc: starting...
W20260619 03:01:54.607028 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:54 dist-test-slave-2gms krb5kdc[13127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:54.835376 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:44469
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:44469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:55.143778 13140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:55.144080 13140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:55.144248 13140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:55.151145 13140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:55.151288 13140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:55.151377 13140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:55.151481 13140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:55.151563 13140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:55.160745 13140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:44469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:44469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:55.161973 13140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:55.163203 13140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:55.174705 13145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:55.174903 13148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:55.174947 13146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:55.175869 13140 server_base.cc:1034] running on GCE node
I20260619 03:01:55.176573 13140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:55.177866 13140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:55.179101 13140 hybrid_clock.cc:648] HybridClock initialized: now 1781838115179058 us; error 49 us; skew 500 ppm
I20260619 03:01:55.179431 13140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:55 dist-test-slave-2gms krb5kdc[13127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838115, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:55.182300 13140 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:55.183712 13140 webserver.cc:460] Webserver started at http://127.22.230.62:33767/ using document root <none> and password file <none>
I20260619 03:01:55.184408 13140 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:55.184548 13140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:55.184831 13140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:55.187595 13140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "e58da0f3527b47279f3ab45a0598d373"
format_stamp: "Formatted at 2026-06-19 03:01:55 on dist-test-slave-2gms"
server_key: "2a5a027d645e05f183d31fd7d6ae9a61"
server_key_iv: "a6fa72a9bb4f659ba4b6500fb02781e0"
server_key_version: "encryptionkey@0"
I20260619 03:01:55.188293 13140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "e58da0f3527b47279f3ab45a0598d373"
format_stamp: "Formatted at 2026-06-19 03:01:55 on dist-test-slave-2gms"
server_key: "2a5a027d645e05f183d31fd7d6ae9a61"
server_key_iv: "a6fa72a9bb4f659ba4b6500fb02781e0"
server_key_version: "encryptionkey@0"
I20260619 03:01:55.192821 13140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:01:55.195858 13155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:55.196985 13140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:01:55.197160 13140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "e58da0f3527b47279f3ab45a0598d373"
format_stamp: "Formatted at 2026-06-19 03:01:55 on dist-test-slave-2gms"
server_key: "2a5a027d645e05f183d31fd7d6ae9a61"
server_key_iv: "a6fa72a9bb4f659ba4b6500fb02781e0"
server_key_version: "encryptionkey@0"
I20260619 03:01:55.197338 13140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:55.225677 13140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:55.229279 13140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:55.229540 13140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:55.246934 13140 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:44469
I20260619 03:01:55.246956 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:44469 every 8 connection(s)
I20260619 03:01:55.248117 13140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:55.251485 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:55.255559 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13140
I20260619 03:01:55.255748 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:55.256275 23448 external_mini_cluster.cc:1428] Setting key 007028574e742fdba9f935fdfc84b04b
I20260619 03:01:55.262147 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373: Bootstrap starting.
I20260619 03:01:55.265499 13207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:55.266681 13207 log.cc:824] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:55.269351 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:55.257867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57757 (local address 127.22.230.62:44469)
0619 03:01:55.258648 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:55.258659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:55.258707 (+ 48us) server_negotiation.cc:407] Connection header received
0619 03:01:55.260831 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:55.260851 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:55.261184 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:55.261470 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:55.262034 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:55.263452 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:55.265250 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:55.265932 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:55.266054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:55.267329 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:55.267338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:55.267351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:55.267842 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:55.267923 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:55.267945 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:55.268081 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:55.268302 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 03:01:55.268536 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":352,"threads_started":1}
I20260619 03:01:55.269402 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373: No bootstrap required, opened a new log
I20260619 03:01:55.275125 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:55.279851 13207 raft_consensus.cc:348] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58da0f3527b47279f3ab45a0598d373" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44469 } }
I20260619 03:01:55.280202 13207 raft_consensus.cc:374] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:55.280326 13207 raft_consensus.cc:729] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58da0f3527b47279f3ab45a0598d373, State: Initialized, Role: FOLLOWER
I20260619 03:01:55.280910 13207 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58da0f3527b47279f3ab45a0598d373" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44469 } }
I20260619 03:01:55.281205 13207 raft_consensus.cc:388] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:55.281361 13207 raft_consensus.cc:482] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:55.281549 13207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:55.283771 13207 raft_consensus.cc:504] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58da0f3527b47279f3ab45a0598d373" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44469 } }
I20260619 03:01:55.284209 13207 leader_election.cc:302] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e58da0f3527b47279f3ab45a0598d373; no voters:
I20260619 03:01:55.284925 13207 leader_election.cc:288] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:55.285075 13217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:55.286911 13207 sys_catalog.cc:564] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:55.288067 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:55.280760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33467 (local address 127.22.230.62:44469)
0619 03:01:55.280944 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:55.280952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:55.281364 (+ 412us) server_negotiation.cc:407] Connection header received
0619 03:01:55.281576 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:55.281579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:55.281654 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:55.281748 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:55.282251 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:55.283053 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:55.284366 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:55.285008 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:55.286040 (+ 1032us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:55.286688 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:55.286697 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:55.286723 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:55.286747 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:55.286795 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:55.286819 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:55.287432 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:55.287492 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 03:01:55.287602 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:01:55.288595 13217 raft_consensus.cc:686] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [term 1 LEADER]: Becoming Leader. State: Replica: e58da0f3527b47279f3ab45a0598d373, State: Running, Role: LEADER
I20260619 03:01:55.289261 13217 consensus_queue.cc:237] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58da0f3527b47279f3ab45a0598d373" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44469 } }
I20260619 03:01:55.293125 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e58da0f3527b47279f3ab45a0598d373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58da0f3527b47279f3ab45a0598d373" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44469 } } }
I20260619 03:01:55.293500 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:55.293902 13219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e58da0f3527b47279f3ab45a0598d373. Latest consensus state: current_term: 1 leader_uuid: "e58da0f3527b47279f3ab45a0598d373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58da0f3527b47279f3ab45a0598d373" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44469 } } }
I20260619 03:01:55.294179 13219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:55.294417 13228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:55.298476 13228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:55.306380 13228 catalog_manager.cc:1347] Generated new cluster ID: 412847e9d25743b0ad2611b9718f67e1
I20260619 03:01:55.306478 13228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:55.333710 13228 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:55.334935 13228 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:55.342275 13228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e58da0f3527b47279f3ab45a0598d373: Generated new TSK 0
I20260619 03:01:55.343006 13228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:55.352124 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13140
W20260619 03:01:55.353464 13202 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57757
W20260619 03:01:55.354122 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13140: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:55.354261 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13140
2026-06-19T03:01:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:55 dist-test-slave-2gms krb5kdc[13238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:55 dist-test-slave-2gms krb5kdc[13238](info): set up 2 sockets
Jun 19 03:01:55 dist-test-slave-2gms krb5kdc[13238](info): commencing operation
krb5kdc: starting...
W20260619 03:01:56.439013 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:56 dist-test-slave-2gms krb5kdc[13238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:56.675156 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:39093
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:56.983515 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:56.983763 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:56.983929 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:56.990906 13251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:56.991017 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:56.991135 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:56.991251 13251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:56.991328 13251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:57.000336 13251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:57.001461 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:57.002710 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:57.013893 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:57.014003 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:57.013954 13256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:57.014815 13251 server_base.cc:1034] running on GCE node
I20260619 03:01:57.015481 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:57.016727 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:57.017957 13251 hybrid_clock.cc:648] HybridClock initialized: now 1781838117017917 us; error 53 us; skew 500 ppm
I20260619 03:01:57.018307 13251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:57 dist-test-slave-2gms krb5kdc[13238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838117, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:57.021355 13251 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:57.022735 13251 webserver.cc:460] Webserver started at http://127.22.230.62:46065/ using document root <none> and password file <none>
I20260619 03:01:57.023314 13251 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:57.023408 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:57.023633 13251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:57.026314 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "ed5ca1eff4af47e685a0bb720457f97b"
format_stamp: "Formatted at 2026-06-19 03:01:57 on dist-test-slave-2gms"
server_key: "2cdbd221f11dd0a3c5581ae59dcc0a8a"
server_key_iv: "6440154bffd55e3a651cab31c38fc4bd"
server_key_version: "encryptionkey@0"
I20260619 03:01:57.027050 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "ed5ca1eff4af47e685a0bb720457f97b"
format_stamp: "Formatted at 2026-06-19 03:01:57 on dist-test-slave-2gms"
server_key: "2cdbd221f11dd0a3c5581ae59dcc0a8a"
server_key_iv: "6440154bffd55e3a651cab31c38fc4bd"
server_key_version: "encryptionkey@0"
I20260619 03:01:57.031627 13251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:01:57.034703 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:57.035851 13251 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 03:01:57.036011 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "ed5ca1eff4af47e685a0bb720457f97b"
format_stamp: "Formatted at 2026-06-19 03:01:57 on dist-test-slave-2gms"
server_key: "2cdbd221f11dd0a3c5581ae59dcc0a8a"
server_key_iv: "6440154bffd55e3a651cab31c38fc4bd"
server_key_version: "encryptionkey@0"
I20260619 03:01:57.036190 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:57.041523 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:57.045580 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:57.045866 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:57.063771 13251 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:39093
I20260619 03:01:57.063812 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:39093 every 8 connection(s)
I20260619 03:01:57.065037 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:57.065364 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13251
I20260619 03:01:57.065567 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:57.066154 23448 external_mini_cluster.cc:1428] Setting key 06f1f80bdb37fa89ef7230cfb7e620a0
I20260619 03:01:57.069155 13318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:57.073321 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:57.067891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48013 (local address 127.22.230.62:39093)
0619 03:01:57.068655 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:57.068665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:57.068705 (+ 40us) server_negotiation.cc:407] Connection header received
0619 03:01:57.070297 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:57.070315 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:57.070668 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:57.070962 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:57.071913 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:57.071919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:57.071929 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:57.071942 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:57.072017 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:57.072035 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:57.072200 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:57.072303 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 03:01:57.072480 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":371,"threads_started":1}
I20260619 03:01:57.076653 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:57.081285 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b: Bootstrap starting.
I20260619 03:01:57.083868 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:57.081609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46965 (local address 127.22.230.62:39093)
0619 03:01:57.081760 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:57.081765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:57.082146 (+ 381us) server_negotiation.cc:407] Connection header received
0619 03:01:57.082248 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:57.082253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:57.082324 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:57.082426 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:57.083274 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:57.083279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:57.083295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 03:01:57.083312 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:57.083355 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 03:01:57.083375 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:57.083493 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:57.083558 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 03:01:57.083606 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 03:01:57.084838 13318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:57.085793 13318 log.cc:824] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:57.088361 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b: No bootstrap required, opened a new log
I20260619 03:01:57.095077 13318 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ca1eff4af47e685a0bb720457f97b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39093 } }
I20260619 03:01:57.095403 13318 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 03:01:57.095511 13318 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5ca1eff4af47e685a0bb720457f97b, State: Initialized, Role: FOLLOWER
I20260619 03:01:57.096011 13318 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ca1eff4af47e685a0bb720457f97b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39093 } }
I20260619 03:01:57.096225 13318 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 03:01:57.096370 13318 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 03:01:57.096530 13318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:57.098517 13318 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ca1eff4af47e685a0bb720457f97b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39093 } }
I20260619 03:01:57.099056 13318 leader_election.cc:302] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5ca1eff4af47e685a0bb720457f97b; no voters:
I20260619 03:01:57.099679 13318 leader_election.cc:288] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 03:01:57.099789 13328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:57.100469 13328 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [term 1 LEADER]: Becoming Leader. State: Replica: ed5ca1eff4af47e685a0bb720457f97b, State: Running, Role: LEADER
I20260619 03:01:57.101037 13328 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ca1eff4af47e685a0bb720457f97b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39093 } }
I20260619 03:01:57.101747 13318 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:57.103910 13329 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5ca1eff4af47e685a0bb720457f97b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ca1eff4af47e685a0bb720457f97b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39093 } } }
I20260619 03:01:57.103935 13330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5ca1eff4af47e685a0bb720457f97b. Latest consensus state: current_term: 1 leader_uuid: "ed5ca1eff4af47e685a0bb720457f97b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5ca1eff4af47e685a0bb720457f97b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 39093 } } }
I20260619 03:01:57.104403 13329 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:57.104465 13330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:57.106154 13335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:57.110911 13335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:01:57.119396 13335 catalog_manager.cc:1347] Generated new cluster ID: bce08ad7e61e46319bf22a31f389b79e
I20260619 03:01:57.119513 13335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:01:57.127692 13335 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:01:57.128051 13335 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:01:57.140635 13335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5ca1eff4af47e685a0bb720457f97b: Generated new TSK 0
I20260619 03:01:57.141532 13335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:01:57.150647 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13251
W20260619 03:01:57.151558 13313 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48013
W20260619 03:01:57.152159 23448 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 03:01:57.152302 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13251
2026-06-19T03:01:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1795 ms)
[----------] 18 tests from AuthTokenIssuingTest (33127 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:01:57 dist-test-slave-2gms krb5kdc[13349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:01:57 dist-test-slave-2gms krb5kdc[13349](info): set up 2 sockets
Jun 19 03:01:57 dist-test-slave-2gms krb5kdc[13349](info): commencing operation
krb5kdc: starting...
W20260619 03:01:58.223018 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:01:58 dist-test-slave-2gms krb5kdc[13349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:01:58.451340 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:44515
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:44515,127.22.230.61:43949,127.22.230.60:42937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:01:58.759218 13362 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 03:01:58.759521 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:58.759631 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:58.759773 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:58.766788 13362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:58.766871 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:58.766983 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:58.767061 13362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:58.767165 13362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:58.776647 13362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:44515,127.22.230.61:43949,127.22.230.60:42937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:58.777797 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:58.779038 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:58.790144 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:58.790273 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:58.790211 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:58.790758 13362 server_base.cc:1034] running on GCE node
I20260619 03:01:58.791589 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:58.792845 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:58.794085 13362 hybrid_clock.cc:648] HybridClock initialized: now 1781838118794006 us; error 53 us; skew 500 ppm
I20260619 03:01:58.794421 13362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:58 dist-test-slave-2gms krb5kdc[13349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:58.800643 13362 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:01:58.801848 13362 webserver.cc:460] Webserver started at http://127.22.230.62:45925/ using document root <none> and password file <none>
I20260619 03:01:58.802476 13362 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:58.802560 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:58.802891 13362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:58.805588 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7"
format_stamp: "Formatted at 2026-06-19 03:01:58 on dist-test-slave-2gms"
server_key: "56b5f2a71060a8cf748811021a55ec12"
server_key_iv: "3b588c342a19109041f2f66fe8b9614c"
server_key_version: "encryptionkey@0"
I20260619 03:01:58.806275 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7"
format_stamp: "Formatted at 2026-06-19 03:01:58 on dist-test-slave-2gms"
server_key: "56b5f2a71060a8cf748811021a55ec12"
server_key_iv: "3b588c342a19109041f2f66fe8b9614c"
server_key_version: "encryptionkey@0"
I20260619 03:01:58.810982 13362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 03:01:58.814005 13377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:58.815205 13362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 03:01:58.815376 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7"
format_stamp: "Formatted at 2026-06-19 03:01:58 on dist-test-slave-2gms"
server_key: "56b5f2a71060a8cf748811021a55ec12"
server_key_iv: "3b588c342a19109041f2f66fe8b9614c"
server_key_version: "encryptionkey@0"
I20260619 03:01:58.815565 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:58.835038 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:58.838373 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:58.838668 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:58.856055 13362 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:44515
I20260619 03:01:58.856086 13428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:44515 every 8 connection(s)
I20260619 03:01:58.857257 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:01:58.860488 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:58.863835 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13362
I20260619 03:01:58.864042 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:01:58.864652 23448 external_mini_cluster.cc:1428] Setting key 7c9fd88d3a4a82e55ea23b28307fc638
I20260619 03:01:58.866518 13429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 03:01:58 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:58.889777 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:58.869351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.22.230.62:44515)
0619 03:01:58.871080 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:58.871086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:58.871122 (+ 36us) server_negotiation.cc:407] Connection header received
0619 03:01:58.873226 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:58.873231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:58.873424 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:58.873572 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:58.875129 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:58.876805 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:58.878121 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:58.879112 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:58.879343 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:58.883199 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:58.883206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:58.883216 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:58.883246 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:58.885890 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:58.886797 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:58.886802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:58.886804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:58.886937 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:58.887361 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:58.887364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:58.887366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:58.887776 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:58.887935 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:58.888571 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 03:01:58.888819 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":1561,"spinlock_wait_cycles":21760,"thread_start_us":1344,"threads_started":1}
I20260619 03:01:58.889865 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:58.869351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:44515 (local address 127.0.0.1:37392)
0619 03:01:58.870181 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:58.870252 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:58.872900 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:58.873644 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:58.873653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:58.874409 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:58.874923 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:58.874948 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:58.877019 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:58.877023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:58.877985 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:58.877995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:58.879375 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:58.879389 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:58.879606 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:58.880370 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:58.880394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:58.883010 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:58.886013 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:58.886026 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:58.886038 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:58.886639 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:58.887046 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:58.887054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:58.887060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:58.887254 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:58.887928 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:58.887939 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:58.888165 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 03:01:58.888424 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 03:01:58.888999 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":109,"thread_start_us":340,"threads_started":1}
I20260619 03:01:58.892510 13429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.230.61@KRBTEST.COM; defaulting to no policy
I20260619 03:01:58.893539 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:58.893183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42780)
0619 03:01:58.893344 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:58.893443 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 03:01:58.893784 13378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
I20260619 03:01:58.895254 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:58.894926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42794)
0619 03:01:58.895096 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:58.895156 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.61@KRBTEST.COM" created.
W20260619 03:01:58.895458 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:43949: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab.
Entry for principal kudu/127.22.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab.
I20260619 03:01:58.923327 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:43949
--webserver_interface=127.22.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab
--principal=kudu/127.22.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:44515,127.22.230.61:43949,127.22.230.60:42937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:01:58.944764 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } attempt: 1
I20260619 03:01:58.945896 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:58.945363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42804)
0619 03:01:58.945577 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:58.945723 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 03:01:58.946877 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:58.946481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42806)
0619 03:01:58.946632 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:58.946726 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 03:01:58.947053 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:43949: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
I20260619 03:01:59.015303 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } attempt: 2
I20260619 03:01:59.016577 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.016024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42810)
0619 03:01:59.016285 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.016403 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 03:01:59.017602 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.017192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42824)
0619 03:01:59.017334 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.017439 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 03:01:59.017795 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:43949: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
I20260619 03:01:59.109082 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } attempt: 3
I20260619 03:01:59.110337 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.109758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42838)
0619 03:01:59.110004 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.110131 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 03:01:59.111502 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.111045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42850)
0619 03:01:59.111211 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.111312 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 03:01:59.111717 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:43949: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
W20260619 03:01:59.230317 13436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 03:01:59.230592 13436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:59.230687 13436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:59.230803 13436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:01:59.237537 13436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:59.237612 13436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:59.237709 13436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:59.237793 13436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:59.237908 13436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:59.246841 13436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:44515,127.22.230.61:43949,127.22.230.60:42937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab
--principal=kudu/127.22.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:43949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:59.247915 13436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:59.249091 13436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 03:01:59.254952 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } attempt: 4
I20260619 03:01:59.256017 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.255504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42856)
0619 03:01:59.255794 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.255899 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 03:01:59.256946 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.256613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42864)
0619 03:01:59.256748 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.256832 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 03:01:59.257107 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:43949: Network error: Client connection negotiation failed: client connection to 127.22.230.61:43949: connect: Connection refused (error 111)
W20260619 03:01:59.260216 13442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:59.260249 13441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:59.260316 13444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:59.260975 13436 server_base.cc:1034] running on GCE node
I20260619 03:01:59.261564 13436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:59.262725 13436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:59.263942 13436 hybrid_clock.cc:648] HybridClock initialized: now 1781838119263887 us; error 35 us; skew 500 ppm
I20260619 03:01:59.264175 13436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:59.267194 13436 init.cc:377] Logged in from keytab as kudu/127.22.230.61@KRBTEST.COM (short username kudu)
I20260619 03:01:59.268498 13436 webserver.cc:460] Webserver started at http://127.22.230.61:34211/ using document root <none> and password file <none>
I20260619 03:01:59.269080 13436 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:59.269162 13436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:59.269392 13436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:59.272043 13436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data/instance:
uuid: "f7daee8078124c839f847220bc26d8a6"
format_stamp: "Formatted at 2026-06-19 03:01:59 on dist-test-slave-2gms"
server_key: "a132a4341ae4ee5ba43863c17037cbef"
server_key_iv: "8e31fede787667da9008f4863ab97fb3"
server_key_version: "encryptionkey@0"
I20260619 03:01:59.272696 13436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal/instance:
uuid: "f7daee8078124c839f847220bc26d8a6"
format_stamp: "Formatted at 2026-06-19 03:01:59 on dist-test-slave-2gms"
server_key: "a132a4341ae4ee5ba43863c17037cbef"
server_key_iv: "8e31fede787667da9008f4863ab97fb3"
server_key_version: "encryptionkey@0"
I20260619 03:01:59.277120 13436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:01:59.280133 13451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:59.281276 13436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:01:59.281446 13436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal
uuid: "f7daee8078124c839f847220bc26d8a6"
format_stamp: "Formatted at 2026-06-19 03:01:59 on dist-test-slave-2gms"
server_key: "a132a4341ae4ee5ba43863c17037cbef"
server_key_iv: "8e31fede787667da9008f4863ab97fb3"
server_key_version: "encryptionkey@0"
I20260619 03:01:59.281637 13436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:59.294811 13436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:59.298434 13436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:59.298684 13436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:59.315564 13436 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.61:43949
I20260619 03:01:59.315595 13502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.61:43949 every 8 connection(s)
I20260619 03:01:59.316709 13436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
I20260619 03:01:59.320170 13503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:59.323786 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13436
I20260619 03:01:59.323956 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-1/wal/instance
I20260619 03:01:59.324343 23448 external_mini_cluster.cc:1428] Setting key 8b188e1e30cec4718e1249eb5a1de1c5
I20260619 03:01:59.326004 13503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:59.345705 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.328271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.22.230.62:44515)
0619 03:01:59.328447 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.328451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.329638 (+ 1187us) server_negotiation.cc:407] Connection header received
0619 03:01:59.331459 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.331465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.331547 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.331667 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.332975 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.333602 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.335754 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.336157 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.336306 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.339983 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.339988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.339991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.340022 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.342623 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.343481 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.343485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.343487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.343594 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.344035 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.344038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.344040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.344316 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.344429 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.345296 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.345471 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 03:01:59.347144 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.328794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:44515 (local address 127.0.0.1:37400)
0619 03:01:59.329490 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.329529 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.331205 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.331890 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.331897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.332312 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.332804 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.332819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.333752 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.333755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.335616 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.335627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.336361 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.336368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.336574 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.337186 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.337205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.339763 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.342748 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.342758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.342774 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.343355 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.343702 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.343710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.343716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.343931 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.344453 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.344463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.344868 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.345199 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.345727 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":318,"threads_started":1}
I20260619 03:01:59.348901 13503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.60@KRBTEST.COM" created.
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:01:59.363588 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.349756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42866)
0619 03:01:59.349956 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.349971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.350043 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.351051 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.351055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.351464 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.351580 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.351587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.353144 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.353148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.354038 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.354049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.355033 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.355043 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.355248 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.355803 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.355829 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.358252 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.361289 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.361295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.361299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.361635 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.361999 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.362004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.362007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.362129 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.362722 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.362728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.362866 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.363003 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.363234 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 03:01:59.363696 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.350008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.22.230.61:43949)
0619 03:01:59.350696 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.350701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.350723 (+ 22us) server_negotiation.cc:407] Connection header received
0619 03:01:59.350760 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.350765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.350922 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.351066 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.351791 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.353006 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.354147 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.354881 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.354970 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.358453 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.358463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.358473 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.358499 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.361143 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.361759 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.361766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.361769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.361887 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.362234 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.362240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.362243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.362545 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.362799 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.363134 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.363336 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":367,"threads_started":1}
I20260619 03:01:59.365826 13503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:01:59.366855 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.366420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49806)
0619 03:01:59.366598 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.366719 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 03:01:59.367105 13452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
I20260619 03:01:59.368526 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.368227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49812)
0619 03:01:59.368370 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.368438 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 03:01:59.368770 13503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab.
Entry for principal kudu/127.22.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab.
I20260619 03:01:59.383805 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:42937
--webserver_interface=127.22.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab
--principal=kudu/127.22.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:44515,127.22.230.61:43949,127.22.230.60:42937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:01:59.418145 13503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 1
I20260619 03:01:59.419186 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.418748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49816)
0619 03:01:59.418929 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.419023 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 03:01:59.420143 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.419822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49818)
0619 03:01:59.419972 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.420049 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 03:01:59.420370 13503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
I20260619 03:01:59.488605 13503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 2
I20260619 03:01:59.489651 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.489212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49824)
0619 03:01:59.489419 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.489500 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 03:01:59.490715 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.490318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49828)
0619 03:01:59.490479 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.490540 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 03:01:59.490943 13503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
I20260619 03:01:59.556329 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } attempt: 5
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:01:59.569144 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.557141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.22.230.61:43949)
0619 03:01:59.557364 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.557370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.557455 (+ 85us) server_negotiation.cc:407] Connection header received
0619 03:01:59.557675 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.557679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.557755 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.557868 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.558672 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.559373 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.560528 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.560965 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.561097 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.564136 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.564140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.564143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.564175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.566625 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.567266 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.567272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.567275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.567402 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.567768 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.567773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.567777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.568054 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.568206 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.568584 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.568798 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 03:01:59.569155 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.557098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42878)
0619 03:01:59.557343 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.557363 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.557532 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.557929 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.557935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.558326 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.558501 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.558510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.559510 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.559517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.560384 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.560395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.561106 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.561111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.561326 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.561952 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.561970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.563953 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.566757 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.566763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.566767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.567141 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.567513 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.567518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.567521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.567659 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.568169 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.568174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.568334 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.568494 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.568772 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 03:01:59.570482 13429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:01:59.571472 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.571100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49834)
0619 03:01:59.571255 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.571340 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 03:01:59.572328 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.572035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49846)
0619 03:01:59.572157 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.572238 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 03:01:59.572513 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
I20260619 03:01:59.582187 13503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 3
I20260619 03:01:59.583128 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.582748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49848)
0619 03:01:59.582934 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.583008 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 03:01:59.584097 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.583711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49850)
0619 03:01:59.583941 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.584003 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260619 03:01:59.584321 13503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
I20260619 03:01:59.623718 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 1
I20260619 03:01:59.624640 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.624219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49866)
0619 03:01:59.624394 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.624488 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 03:01:59.625542 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.625265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49874)
0619 03:01:59.625391 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.625460 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 03:01:59.625816 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
W20260619 03:01:59.689381 13510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 03:01:59.689672 13510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:01:59.689790 13510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:01:59.689936 13510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 03:01:59.694006 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 2
I20260619 03:01:59.695147 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.694615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49882)
0619 03:01:59.694805 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.694935 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 03:01:59.696019 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.695742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49892)
0619 03:01:59.695860 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.695920 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 03:01:59.696244 13429 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
W20260619 03:01:59.697528 13510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:01:59.697620 13510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:01:59.697722 13510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:01:59.697798 13510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:01:59.697878 13510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:01:59.706931 13510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:44515,127.22.230.61:43949,127.22.230.60:42937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab
--principal=kudu/127.22.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:01:59.707965 13510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:01:59.709084 13510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:01:59.719751 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:59.719784 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:01:59.720197 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:01:59.720907 13510 server_base.cc:1034] running on GCE node
I20260619 03:01:59.721624 13510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:01:59.722819 13510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:01:59.724042 13510 hybrid_clock.cc:648] HybridClock initialized: now 1781838119723997 us; error 40 us; skew 500 ppm
I20260619 03:01:59.724354 13510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:01:59.727244 13510 init.cc:377] Logged in from keytab as kudu/127.22.230.60@KRBTEST.COM (short username kudu)
I20260619 03:01:59.727521 13503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 4
I20260619 03:01:59.728624 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.728089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49906)
0619 03:01:59.728370 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.728463 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 03:01:59.729416 13510 webserver.cc:460] Webserver started at http://127.22.230.60:40551/ using document root <none> and password file <none>
I20260619 03:01:59.729766 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.729354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49920)
0619 03:01:59.729513 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.729595 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 03:01:59.730058 13503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:42937: Network error: Client connection negotiation failed: client connection to 127.22.230.60:42937: connect: Connection refused (error 111)
I20260619 03:01:59.730331 13510 fs_manager.cc:359] Metadata directory not provided
I20260619 03:01:59.730414 13510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:01:59.730724 13510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:01:59.733256 13510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data/instance:
uuid: "2aaea55a6b8744f7903f1dca515f84bd"
format_stamp: "Formatted at 2026-06-19 03:01:59 on dist-test-slave-2gms"
server_key: "85732edf6134a177a85befd48414fbe7"
server_key_iv: "9b50a659cb9caa5eb91bae4ae83d6118"
server_key_version: "encryptionkey@0"
I20260619 03:01:59.733906 13510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal/instance:
uuid: "2aaea55a6b8744f7903f1dca515f84bd"
format_stamp: "Formatted at 2026-06-19 03:01:59 on dist-test-slave-2gms"
server_key: "85732edf6134a177a85befd48414fbe7"
server_key_iv: "9b50a659cb9caa5eb91bae4ae83d6118"
server_key_version: "encryptionkey@0"
I20260619 03:01:59.738718 13510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 03:01:59.741752 13525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:01:59.743110 13510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 03:01:59.743229 13510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal
uuid: "2aaea55a6b8744f7903f1dca515f84bd"
format_stamp: "Formatted at 2026-06-19 03:01:59 on dist-test-slave-2gms"
server_key: "85732edf6134a177a85befd48414fbe7"
server_key_iv: "9b50a659cb9caa5eb91bae4ae83d6118"
server_key_version: "encryptionkey@0"
I20260619 03:01:59.743403 13510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:01:59.764436 13510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:01:59.768039 13510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:01:59.768230 13510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:01:59.785539 13510 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.60:42937
I20260619 03:01:59.785544 13576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.60:42937 every 8 connection(s)
I20260619 03:01:59.786846 13510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
I20260619 03:01:59.790174 13577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:01:59.794888 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13510
I20260619 03:01:59.795089 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781837774831470-23448-0/minicluster-data/master-2/wal/instance
I20260619 03:01:59.795521 23448 external_mini_cluster.cc:1428] Setting key af5904f54b1e8b5d8271c5feae3ed1cd
I20260619 03:01:59.796624 13577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:01:59.802464 13429 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 3
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:01:59.816835 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.799073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.22.230.62:44515)
0619 03:01:59.799240 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.799244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.799944 (+ 700us) server_negotiation.cc:407] Connection header received
0619 03:01:59.800644 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.800652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.800736 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.800850 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.802086 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.803104 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.805477 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.806112 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.806248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.810336 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.810342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.810347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.810388 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.813478 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.814258 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.814267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.814272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.814436 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.814871 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.814878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.814884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.815284 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.815452 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.816233 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.816468 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":67}
I20260619 03:01:59.817633 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.799174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:44515 (local address 127.0.0.1:37402)
0619 03:01:59.799805 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.799841 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.800373 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.800938 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.800944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.801574 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.801921 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.801937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.803315 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.803320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.805327 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.805349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.806358 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.806364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.806560 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.807456 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.807489 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.810127 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.813612 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.813627 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.813635 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.814127 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.814543 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.814547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.814551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.814758 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.815438 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.815446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.815830 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.816131 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.816528 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":313,"threads_started":1}
I20260619 03:01:59.818775 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.803050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49952)
0619 03:01:59.803334 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.803350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.803438 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.804296 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.804303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.804705 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.804842 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.804849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.806254 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.806262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.807716 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.807732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.808842 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.808849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.809054 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.809660 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.809677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.812180 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.815980 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.815987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.815993 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.816492 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.816926 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.816932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.816937 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.817099 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.817706 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.817714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.817904 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.818024 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.818335 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":11}
I20260619 03:01:59.819427 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.797256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.22.230.60:42937)
0619 03:01:59.798017 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.798028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.798063 (+ 35us) server_negotiation.cc:407] Connection header received
0619 03:01:59.800361 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.800385 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.800594 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.800741 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.802141 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.803900 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.805628 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.806139 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.806907 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.810907 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.810920 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.810936 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.810975 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.814831 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.815647 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.815654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.815657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.815796 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.816212 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.816217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.816220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.816605 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.818548 (+ 1943us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.818726 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.818958 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260619 03:01:59.819399 13577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:01:59.820182 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.803315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.22.230.60:42937)
0619 03:01:59.804069 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.804072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.804086 (+ 14us) server_negotiation.cc:407] Connection header received
0619 03:01:59.804115 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.804118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.804160 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.804275 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.804989 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.806113 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.807876 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.808645 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.808784 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.812440 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.812448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.812458 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.812497 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.815824 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.816629 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.816633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.816637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.816769 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.817235 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.817239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.817242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.817562 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.819125 (+ 1563us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.819248 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.819485 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":415,"threads_started":1}
I20260619 03:01:59.826315 13429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7: Bootstrap starting.
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:01:59.829538 13429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7: Neither blocks nor log segments found. Creating new log.
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:01:59.830781 13429 log.cc:824] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:59.833691 13429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7: No bootstrap required, opened a new log
I20260619 03:01:59.835707 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.820502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:43949 (local address 127.0.0.1:42892)
0619 03:01:59.820931 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.820946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.821040 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.821338 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.821342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.821797 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.821959 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.821968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.823219 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.823223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.824480 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.824494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.825666 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.825671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.825843 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.826339 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.826361 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.828844 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.833046 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.833052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.833056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.833520 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.833964 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.833968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.833971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.834114 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.834769 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.834773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.834915 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.835093 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.835338 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260619 03:01:59.835816 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.820624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.22.230.61:43949)
0619 03:01:59.820806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.820811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.821030 (+ 219us) server_negotiation.cc:407] Connection header received
0619 03:01:59.821147 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.821152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.821223 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.821308 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.822120 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.823058 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.824622 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.825530 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.825636 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.829061 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.829068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.829073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.829132 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.832952 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.833642 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.833647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.833650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.833852 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.834231 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.834237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.834241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.834552 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.834711 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.835188 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.835430 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 03:01:59.836962 13577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:01:59.837417 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.822233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.22.230.62:44515)
0619 03:01:59.822364 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.822379 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.822439 (+ 60us) server_negotiation.cc:407] Connection header received
0619 03:01:59.822620 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.822625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.822682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.822769 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.823725 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.824535 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.826267 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.826776 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.826905 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.831347 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.831351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.831354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.831408 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.834513 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.835050 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.835056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.835059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.835230 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.835662 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.835667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.835670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.836065 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.836344 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.837002 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.837176 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 03:01:59.844030 13429 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:01:59.844326 13429 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4a3bfcb8df4a7ca0de48aa664a2ff7, State: Initialized, Role: FOLLOWER
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:01:59.844794 13429 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:01:59.845633 13584 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:01:59.846124 13584 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 03:01:59 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:01:59.846909 13429 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:59.851619 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.837522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49954)
0619 03:01:59.837906 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 03:01:59.837922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 03:01:59.838014 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:01:59.838335 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:01:59.838339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:01:59.838754 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:01:59.838917 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.838926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.840198 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.840202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.841489 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:01:59.841503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.842287 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.842294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:01:59.842472 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.842938 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:01:59.842958 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:01:59.845202 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:01:59.849056 (+ 3854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.849060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.849063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.849565 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.849901 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:01:59.849908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:01:59.849912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:01:59.850054 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:01:59.850761 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:01:59.850765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:01:59.850908 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 03:01:59.851040 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 03:01:59.851292 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291}
I20260619 03:01:59.852370 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.837606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.22.230.60:42937)
0619 03:01:59.837755 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.837761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.837994 (+ 233us) server_negotiation.cc:407] Connection header received
0619 03:01:59.838096 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.838103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.838187 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.838291 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.839042 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.840054 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.841628 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.842153 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.842254 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.845396 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.845403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.845408 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.845440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.848915 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.849674 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.849682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.849685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.849797 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.850175 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.850181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.850186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.850568 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.850716 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.851906 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.852089 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 03:01:59.854123 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.839065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.22.230.61:43949)
0619 03:01:59.839235 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.839240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.839254 (+ 14us) server_negotiation.cc:407] Connection header received
0619 03:01:59.839297 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.839300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.839343 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.839440 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.840535 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.841230 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.842625 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.843097 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.843217 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.847926 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.847935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.847942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.847982 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.851677 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.852402 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.852408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.852412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.852553 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.853040 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.853045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.853048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.853270 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.853436 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.853766 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.853909 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 03:01:59.857234 13577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: Bootstrap starting.
I20260619 03:01:59.860288 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:01:59.863550 13404 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37416
I20260619 03:01:59.863925 13404 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 03:01:59.864114 13404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:59.867486 13404 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:01:59.869381 13404 leader_election.cc:288] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [CANDIDATE]: Term 1 election: Requested vote from peers f7daee8078124c839f847220bc26d8a6 (127.22.230.61:43949), 2aaea55a6b8744f7903f1dca515f84bd (127.22.230.60:42937)
W20260619 03:01:59.876667 13595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 03:01:59.876793 13595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 03:01:59.879386 13577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: Neither blocks nor log segments found. Creating new log.
I20260619 03:01:59.880944 13577 log.cc:824] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: Log is configured to *not* fsync() on all Append() calls
I20260619 03:01:59.862852 13362 master_runner.cc:186] Error getting master registration for 127.22.230.61:43949: OK, instance_id { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" instance_seqno: 1781838119310915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 03:01:59.882381 13362 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 03:01:59.885242 13577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: No bootstrap required, opened a new log
I20260619 03:01:59.905680 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.885576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.22.230.61:43949)
0619 03:01:59.885961 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.885967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.885982 (+ 15us) server_negotiation.cc:407] Connection header received
0619 03:01:59.886035 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.886038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.886090 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.886201 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.888489 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.889939 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.892444 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.893096 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.893207 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.897220 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.897229 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.897236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.897288 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.901472 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.902642 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.902650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.902653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.902844 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.903407 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.903415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.903420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.903897 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.904199 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.904898 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.905195 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260619 03:01:59.907572 13577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:01:59.907994 13577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aaea55a6b8744f7903f1dca515f84bd, State: Initialized, Role: FOLLOWER
I20260619 03:01:59.908483 13577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:01:59.909608 13605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:01:59.909991 13605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:01:59.916819 13577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:01:59.916875 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2aaea55a6b8744f7903f1dca515f84bd"
I20260619 03:01:59.917351 13552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:01:59.920989 13552 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4a3bfcb8df4a7ca0de48aa664a2ff7 in term 1.
I20260619 03:01:59.921902 13378 leader_election.cc:302] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aaea55a6b8744f7903f1dca515f84bd, bc4a3bfcb8df4a7ca0de48aa664a2ff7; no voters:
I20260619 03:01:59.922396 13584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:01:59.932572 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.885761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.22.230.62:44515)
0619 03:01:59.886485 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.886490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.906471 (+ 19981us) server_negotiation.cc:407] Connection header received
0619 03:01:59.910745 (+ 4274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.910751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.910823 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.910889 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.912609 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.913639 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.915321 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.916045 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.916171 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.921292 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.921298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.921302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.921344 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.926280 (+ 4936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.927898 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.927906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.927912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.928151 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.929026 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.929034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.929039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.929521 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.929823 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.931042 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.931623 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260619 03:01:59.932621 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:01:59.887423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.22.230.60:42937)
0619 03:01:59.887597 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 03:01:59.887607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:01:59.902841 (+ 15234us) server_negotiation.cc:407] Connection header received
0619 03:01:59.903588 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:01:59.903594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:01:59.903662 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:01:59.903746 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:01:59.910557 (+ 6811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.911605 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.913435 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:01:59.913927 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:01:59.914063 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:01:59.918663 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:01:59.918670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:01:59.918675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:01:59.918708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:01:59.922208 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.929617 (+ 7409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.929624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.929627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.929816 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:01:59.930678 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:01:59.930685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:01:59.930690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:01:59.931025 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:01:59.931276 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 03:01:59.932057 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 03:01:59.932292 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:01:59.936076 13584 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [term 1 LEADER]: Becoming Leader. State: Replica: bc4a3bfcb8df4a7ca0de48aa664a2ff7, State: Running, Role: LEADER
I20260619 03:01:59.936681 13584 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:01:59.947190 13607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4a3bfcb8df4a7ca0de48aa664a2ff7. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:01:59.947717 13607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: This master's current role is: LEADER
I20260619 03:01:59.948958 13617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:01:59.952380 13617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 03:01:59.969585 13619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 03:01:59.969724 13619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 03:01:59.971325 13552 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [term 1 FOLLOWER]: Refusing update from remote peer bc4a3bfcb8df4a7ca0de48aa664a2ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:01:59.973680 13607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:01:59.961076 13510 master_runner.cc:186] Error getting master registration for 127.22.230.61:43949: OK, instance_id { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" instance_seqno: 1781838119310915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 03:01:59.974279 13510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 03:01:59.985715 13605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4a3bfcb8df4a7ca0de48aa664a2ff7. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:01:59.986007 13605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:00.007251 13605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:02:00.007524 13605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:00.009639 13607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:02:00.009917 13607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: This master's current role is: LEADER
I20260619 03:02:00.010488 13617 catalog_manager.cc:1347] Generated new cluster ID: af39f289c0ce4845907b708330296ce1
I20260619 03:02:00.010607 13617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:02:00.029242 13503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } attempt: 5
I20260619 03:02:00.040421 13617 catalog_manager.cc:1370] Generated new certificate authority record
Jun 19 03:02:00 dist-test-slave-2gms krb5kdc[13349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:02:00.043391 13617 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:02:00.053473 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.030373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.22.230.60:42937)
0619 03:02:00.030549 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.030555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.030614 (+ 59us) server_negotiation.cc:407] Connection header received
0619 03:02:00.030668 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.030673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.030753 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.030845 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.032356 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.033452 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.036069 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.036728 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.036843 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.042008 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:00.042019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:00.042024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:00.042059 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:00.045898 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:00.049039 (+ 3141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:00.049078 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:00.049081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:00.049381 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:00.050073 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:00.050078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:00.050081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:00.050666 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:00.050884 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.051633 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.052056 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 03:02:00.054353 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.029739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:42937 (local address 127.0.0.1:49974)
0619 03:02:00.030060 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:00.030079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:00.030181 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:00.030920 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:00.030924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:00.031438 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:00.031642 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:00.031654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.033606 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.033611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:00.035124 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:00.035140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.037069 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.037075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:00.037293 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.038019 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:00.038044 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:00.041749 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:00.046138 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:00.046143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:00.046147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:00.047002 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:00.049632 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:00.049637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:00.049641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:00.049902 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:00.051006 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:00.051011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:00.051241 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 03:02:00.051976 (+ 735us) client_negotiation.cc:240] Negotiation successful
0619 03:02:00.052849 (+ 873us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260619 03:02:00.062423 13503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: Bootstrap starting.
I20260619 03:02:00.068966 13617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7: Generated new TSK 0
I20260619 03:02:00.069744 13503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: Neither blocks nor log segments found. Creating new log.
I20260619 03:02:00.070719 13617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 03:02:00.071704 13503 log.cc:824] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: Log is configured to *not* fsync() on all Append() calls
I20260619 03:02:00.078380 13503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: No bootstrap required, opened a new log
I20260619 03:02:00.097210 13503 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:02:00.097851 13503 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7daee8078124c839f847220bc26d8a6, State: Initialized, Role: FOLLOWER
I20260619 03:02:00.098825 13503 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } }
I20260619 03:02:00.111123 13627 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:02:00.111557 13627 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:00.113646 13478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7daee8078124c839f847220bc26d8a6"
I20260619 03:02:00.114029 13478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:02:00.117726 13478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4a3bfcb8df4a7ca0de48aa664a2ff7 in term 1.
I20260619 03:02:00.118434 13477 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [term 1 FOLLOWER]: Refusing update from remote peer bc4a3bfcb8df4a7ca0de48aa664a2ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:02:00.120218 13584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:02:00.125711 13503 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:02:00.163859 13627 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc4a3bfcb8df4a7ca0de48aa664a2ff7. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:02:00.164144 13627 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:00.164455 13635 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:02:00.164846 13635 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 03:02:00.192721 13507 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.177448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.22.230.61:43949)
0619 03:02:00.177623 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.177629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.189438 (+ 11809us) server_negotiation.cc:407] Connection header received
0619 03:02:00.191299 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.191305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.191399 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.191467 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.192510 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42918: BlockingRecv error: recv got EOF from 127.0.0.1:42918 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 03:02:00.197398 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.189947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49982 (local address 127.22.230.60:42937)
0619 03:02:00.190118 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.190124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.194729 (+ 4605us) server_negotiation.cc:407] Connection header received
0619 03:02:00.196214 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.196221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.196295 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.196370 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.197146 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49982: BlockingRecv error: recv got EOF from 127.0.0.1:49982 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 03:02:00.198530 13432 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.185769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.22.230.62:44515)
0619 03:02:00.186027 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.186035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.194262 (+ 8227us) server_negotiation.cc:407] Connection header received
0619 03:02:00.194963 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.194973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.195053 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.195117 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.198321 (+ 3204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37430: BlockingRecv error: recv got EOF from 127.0.0.1:37430 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 03:02:00.217363 13652 catalog_manager.cc:1259] Loaded cluster ID: af39f289c0ce4845907b708330296ce1
I20260619 03:02:00.217456 13652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: loading cluster ID for follower catalog manager: success
I20260619 03:02:00.221451 13652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: acquiring CA information for follower catalog manager: success
I20260619 03:02:00.223793 13652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7daee8078124c839f847220bc26d8a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 03:02:00.247046 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.229288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.22.230.61:43949)
0619 03:02:00.229604 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.229610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.240166 (+ 10556us) server_negotiation.cc:407] Connection header received
0619 03:02:00.240610 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.240615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.240769 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.240880 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.241529 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.242534 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.244414 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.245039 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.245128 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.245791 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.246288 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.246380 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.246608 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.246713 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
W20260619 03:02:00.308979 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.286095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.22.230.60:42937)
0619 03:02:00.286367 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.286373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.307117 (+ 20744us) server_negotiation.cc:407] Connection header received
0619 03:02:00.307298 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.307304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.307435 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.307541 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.308739 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49998: BlockingRecv error: recv got EOF from 127.0.0.1:49998 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 03:02:00.369481 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.351449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.22.230.61:43949)
0619 03:02:00.351625 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.351632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.363098 (+ 11466us) server_negotiation.cc:407] Connection header received
0619 03:02:00.363302 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.363308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.363421 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.363512 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.364090 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.365221 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.367300 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.368022 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.368127 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.368680 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.368820 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.368891 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.369090 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.369168 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 03:02:00.418370 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.405699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.22.230.60:42937)
0619 03:02:00.405875 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.405880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.416485 (+ 10605us) server_negotiation.cc:407] Connection header received
0619 03:02:00.416963 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.416970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.417074 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.417147 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.418132 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50006: BlockingRecv error: recv got EOF from 127.0.0.1:50006 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:02:00.471937 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.454150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.22.230.61:43949)
0619 03:02:00.454353 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.454360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.464971 (+ 10611us) server_negotiation.cc:407] Connection header received
0619 03:02:00.465488 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.465495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.465591 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.465674 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.466398 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.467580 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.469602 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.470265 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.470367 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.471021 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.471163 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.471224 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.471485 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.471576 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 03:02:00.494212 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.492043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.22.230.60:42937)
0619 03:02:00.492228 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.492234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.492760 (+ 526us) server_negotiation.cc:407] Connection header received
0619 03:02:00.492935 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.492944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.493032 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.493165 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.493988 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50020: BlockingRecv error: recv got EOF from 127.0.0.1:50020 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 03:02:00.498769 13584 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4a3bfcb8df4a7ca0de48aa664a2ff7" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 44515 } } peers { permanent_uuid: "f7daee8078124c839f847220bc26d8a6" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 43949 } } peers { permanent_uuid: "2aaea55a6b8744f7903f1dca515f84bd" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 42937 } } }
I20260619 03:02:00.499064 13584 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4a3bfcb8df4a7ca0de48aa664a2ff7 [sys.catalog]: This master's current role is: LEADER
I20260619 03:02:00.511071 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.504599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.22.230.61:43949)
0619 03:02:00.504771 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.504782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.505273 (+ 491us) server_negotiation.cc:407] Connection header received
0619 03:02:00.505433 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.505438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.505513 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.505602 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.505989 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.507129 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.508990 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.509692 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.509832 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.510334 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.510469 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.510560 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.510751 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.510826 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 03:02:00.519354 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.517191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.22.230.60:42937)
0619 03:02:00.517454 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.517460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.517923 (+ 463us) server_negotiation.cc:407] Connection header received
0619 03:02:00.518071 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.518075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.518147 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.518244 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.519088 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50028: BlockingRecv error: recv got EOF from 127.0.0.1:50028 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260619 03:02:00.536899 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.530300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.22.230.61:43949)
0619 03:02:00.530494 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.530500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.530969 (+ 469us) server_negotiation.cc:407] Connection header received
0619 03:02:00.531163 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.531174 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.531250 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.531337 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.531813 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.532933 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.534782 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.535509 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.535635 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.536107 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.536254 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.536343 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.536516 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.536588 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 03:02:00.549237 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.547328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.22.230.60:42937)
0619 03:02:00.547529 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.547536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.547556 (+ 20us) server_negotiation.cc:407] Connection header received
0619 03:02:00.547791 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.547798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.547898 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.547979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.548973 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50032: BlockingRecv error: recv got EOF from 127.0.0.1:50032 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 03:02:00.575613 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.568999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.22.230.61:43949)
0619 03:02:00.569211 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.569218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.569784 (+ 566us) server_negotiation.cc:407] Connection header received
0619 03:02:00.569978 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.569984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.570090 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.570198 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.570667 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.571830 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.573701 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.574323 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.574445 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.574937 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.575048 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.575136 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.575310 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.575387 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 03:02:00.592438 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.590217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.22.230.60:42937)
0619 03:02:00.590411 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.590417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.590895 (+ 478us) server_negotiation.cc:407] Connection header received
0619 03:02:00.591046 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.591052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.591181 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.591323 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.592158 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50048: BlockingRecv error: recv got EOF from 127.0.0.1:50048 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 03:02:00.633661 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.627695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.22.230.61:43949)
0619 03:02:00.627913 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.627921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.628472 (+ 551us) server_negotiation.cc:407] Connection header received
0619 03:02:00.628659 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.628663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.628725 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.628804 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.629280 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.630024 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.631940 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.632377 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.632471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.632945 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.633031 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.633094 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.633270 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.633327 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 03:02:00.641495 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.639530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.22.230.60:42937)
0619 03:02:00.639727 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.639733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.640198 (+ 465us) server_negotiation.cc:407] Connection header received
0619 03:02:00.640373 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.640377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.640444 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.640521 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.641307 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50054: BlockingRecv error: recv got EOF from 127.0.0.1:50054 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 03:02:00.713248 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.708392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.22.230.61:43949)
0619 03:02:00.708602 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.708610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.708907 (+ 297us) server_negotiation.cc:407] Connection header received
0619 03:02:00.709129 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.709133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.709233 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.709335 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.709739 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.710451 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.711722 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.712099 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.712235 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.712599 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.712694 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.712785 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.712922 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.712997 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 03:02:00.720482 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.718589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.22.230.60:42937)
0619 03:02:00.718808 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.718816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.719112 (+ 296us) server_negotiation.cc:407] Connection header received
0619 03:02:00.719322 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.719326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.719445 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.719566 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.720270 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50066: BlockingRecv error: recv got EOF from 127.0.0.1:50066 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 03:02:00.856704 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:00.851808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.22.230.61:43949)
0619 03:02:00.852039 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.852046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.852354 (+ 308us) server_negotiation.cc:407] Connection header received
0619 03:02:00.852503 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.852507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.852570 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.852654 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:00.853047 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.853694 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.855137 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:00.855538 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:00.855625 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:00.856091 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:00.856177 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:00.856238 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:00.856391 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 03:02:00.856445 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 03:02:00.865782 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:00.864110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.22.230.60:42937)
0619 03:02:00.864297 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:00.864303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:00.864692 (+ 389us) server_negotiation.cc:407] Connection header received
0619 03:02:00.864803 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:00.864809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:00.864888 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:00.864982 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:00.865580 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50078: BlockingRecv error: recv got EOF from 127.0.0.1:50078 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 03:02:00.971616 13619 catalog_manager.cc:1259] Loaded cluster ID: af39f289c0ce4845907b708330296ce1
I20260619 03:02:00.971746 13619 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: loading cluster ID for follower catalog manager: success
I20260619 03:02:00.974125 13619 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: acquiring CA information for follower catalog manager: success
I20260619 03:02:00.975781 13619 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2aaea55a6b8744f7903f1dca515f84bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 03:02:01.130199 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:01.125252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.22.230.61:43949)
0619 03:02:01.125464 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:01.125471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:01.125645 (+ 174us) server_negotiation.cc:407] Connection header received
0619 03:02:01.125918 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:01.125923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:01.125990 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:01.126089 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:01.126504 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.127286 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.128590 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.129019 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.129150 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:01.129530 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:01.129637 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:01.129733 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:01.129881 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 03:02:01.129956 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 03:02:01.141455 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:01.135709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.22.230.60:42937)
0619 03:02:01.135922 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:01.135927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:01.136027 (+ 100us) server_negotiation.cc:407] Connection header received
0619 03:02:01.136221 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:01.136228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:01.136378 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:01.136498 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:01.136887 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.137561 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.138988 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.139673 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.139834 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:01.140360 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:01.140880 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:01.141043 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:01.141095 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 03:02:01.141173 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 03:02:01.154896 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:01.147464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.22.230.61:43949)
0619 03:02:01.147661 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:01.147670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:01.148678 (+ 1008us) server_negotiation.cc:407] Connection header received
0619 03:02:01.148955 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:01.148960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:01.149043 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:01.149106 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:01.150261 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.151154 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.152953 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.153598 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.153717 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:01.154177 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:01.154304 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:01.154385 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:01.154531 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 03:02:01.154625 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 03:02:01.155408 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:01.148930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.22.230.60:42937)
0619 03:02:01.149079 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:01.149086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:01.149109 (+ 23us) server_negotiation.cc:407] Connection header received
0619 03:02:01.149724 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:01.149729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:01.149802 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:01.149953 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:01.150560 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.151510 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.153638 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.154161 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.154285 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:01.154712 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:01.154836 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:01.154934 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:01.155044 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 03:02:01.155112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 03:02:01.156205 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:01.147447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.22.230.62:44515)
0619 03:02:01.148059 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:01.148065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:01.148503 (+ 438us) server_negotiation.cc:407] Connection header received
0619 03:02:01.148753 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:01.148759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:01.148982 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:01.149142 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:01.149869 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.150882 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.152861 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:01.153307 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:01.153445 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:01.153989 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:01.154440 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:01.154538 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:01.155846 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0619 03:02:01.155932 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":347,"threads_started":1}
I20260619 03:02:01.161005 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13362
I20260619 03:02:01.162138 13424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:02:01.269942 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13362
I20260619 03:02:01.291379 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13436
I20260619 03:02:01.292369 13498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:02:01.396745 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13436
I20260619 03:02:01.413856 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13510
I20260619 03:02:01.414898 13572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:02:01.517238 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13510
2026-06-19T03:02:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4372 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 03:02:01 dist-test-slave-2gms krb5kdc[13768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 03:02:01 dist-test-slave-2gms krb5kdc[13768](info): set up 2 sockets
Jun 19 03:02:01 dist-test-slave-2gms krb5kdc[13768](info): commencing operation
krb5kdc: starting...
W20260619 03:02:02.603408 23448 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/test-user.keytab.
Jun 19 03:02:02 dist-test-slave-2gms krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T03:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T03:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
Entry for principal kudu/127.22.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab.
I20260619 03:02:02.853150 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:34335
--webserver_interface=127.22.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:34335,127.22.230.61:34529,127.22.230.60:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 03:02:03.164798 13781 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 03:02:03.165131 13781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:02:03.165246 13781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:02:03.165373 13781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:02:03.172274 13781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:02:03.172385 13781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:02:03.172499 13781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:02:03.172578 13781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:02:03.172662 13781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:02:03.181762 13781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:34335,127.22.230.61:34529,127.22.230.60:36047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.62.keytab
--principal=kudu/127.22.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.62:34335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:02:03.182937 13781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:02:03.184157 13781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:02:03.195122 13786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:02:03.195230 13787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:02:03.195150 13789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:02:03.196139 13781 server_base.cc:1034] running on GCE node
I20260619 03:02:03.196794 13781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:02:03.197988 13781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:02:03.199244 13781 hybrid_clock.cc:648] HybridClock initialized: now 1781838123199176 us; error 22 us; skew 500 ppm
I20260619 03:02:03.199549 13781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:02:03 dist-test-slave-2gms krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:02:03.203037 13781 init.cc:377] Logged in from keytab as kudu/127.22.230.62@KRBTEST.COM (short username kudu)
I20260619 03:02:03.204314 13781 webserver.cc:460] Webserver started at http://127.22.230.62:34741/ using document root <none> and password file <none>
I20260619 03:02:03.204929 13781 fs_manager.cc:359] Metadata directory not provided
I20260619 03:02:03.205021 13781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:02:03.205288 13781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:02:03.208310 13781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data/instance:
uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b"
format_stamp: "Formatted at 2026-06-19 03:02:03 on dist-test-slave-2gms"
server_key: "156bf046b3acc98975528f05f85a695c"
server_key_iv: "bc99aa533189faa5decb121324a769a6"
server_key_version: "encryptionkey@0"
I20260619 03:02:03.209072 13781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal/instance:
uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b"
format_stamp: "Formatted at 2026-06-19 03:02:03 on dist-test-slave-2gms"
server_key: "156bf046b3acc98975528f05f85a695c"
server_key_iv: "bc99aa533189faa5decb121324a769a6"
server_key_version: "encryptionkey@0"
I20260619 03:02:03.214053 13781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 03:02:03.217453 13796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:02:03.218771 13781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 03:02:03.218922 13781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal
uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b"
format_stamp: "Formatted at 2026-06-19 03:02:03 on dist-test-slave-2gms"
server_key: "156bf046b3acc98975528f05f85a695c"
server_key_iv: "bc99aa533189faa5decb121324a769a6"
server_key_version: "encryptionkey@0"
I20260619 03:02:03.219092 13781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:02:03.233948 13781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:02:03.238091 13781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:02:03.238306 13781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:02:03.256865 13781 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.62:34335
I20260619 03:02:03.256891 13847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.62:34335 every 8 connection(s)
I20260619 03:02:03.258132 13781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/data/info.pb
I20260619 03:02:03.261461 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:02:03.263556 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13781
I20260619 03:02:03.263758 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-0/wal/instance
I20260619 03:02:03.264246 23448 external_mini_cluster.cc:1428] Setting key 3f41da6c9986e3a35f78a52fd2704376
I20260619 03:02:03.270344 13848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 03:02:03 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:02:03.293843 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.273810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34335 (local address 127.0.0.1:49954)
0619 03:02:03.274632 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.274685 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:03.276939 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:03.277714 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:03.277719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:03.278215 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:03.278508 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.278524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.280856 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.280860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.281862 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.281873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.283143 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.283147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.283253 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.283859 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:03.283879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:03.286052 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:03.290219 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.290228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.290235 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.290741 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.291170 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.291173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.291175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.291319 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.292258 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:03.292264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:03.292441 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 03:02:03.292609 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 03:02:03.292910 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260619 03:02:03.293886 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.273810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.22.230.62:34335)
0619 03:02:03.275517 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:03.275523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:03.275580 (+ 57us) server_negotiation.cc:407] Connection header received
0619 03:02:03.277282 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:03.277292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:03.277529 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:03.277675 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:03.278744 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.280718 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.281996 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.282951 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.283120 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.286279 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:03.286291 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:03.286306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:03.286345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:03.290099 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.290867 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.290877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.290882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.291061 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.291440 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.291447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.291453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.292122 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:03.292330 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:03.292790 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 03:02:03.293058 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1543,"spinlock_wait_cycles":21376,"thread_start_us":1322,"threads_started":1}
WARNING: no policy specified for kudu/127.22.230.61@KRBTEST.COM; defaulting to no policy
I20260619 03:02:03.296350 13848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:02:03.297344 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.296879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58536)
0619 03:02:03.297018 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.297108 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.61@KRBTEST.COM" created.
W20260619 03:02:03.297582 13800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
I20260619 03:02:03.299139 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.298714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58546)
0619 03:02:03.298881 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.298970 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 03:02:03.299327 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:34529: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab.
Entry for principal kudu/127.22.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab.
I20260619 03:02:03.327551 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:34529
--webserver_interface=127.22.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab
--principal=kudu/127.22.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:34335,127.22.230.61:34529,127.22.230.60:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:02:03.348670 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } attempt: 1
I20260619 03:02:03.349833 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.349412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58550)
0619 03:02:03.349597 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.349698 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 03:02:03.350842 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.350491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58564)
0619 03:02:03.350658 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.350725 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 03:02:03.351011 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:34529: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
I20260619 03:02:03.419236 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } attempt: 2
I20260619 03:02:03.420343 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.419876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58566)
0619 03:02:03.420124 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.420208 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 03:02:03.421376 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.421027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58580)
0619 03:02:03.421174 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.421244 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 03:02:03.421556 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:34529: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
I20260619 03:02:03.512799 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } attempt: 3
I20260619 03:02:03.514084 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.513610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58594)
0619 03:02:03.513858 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.513950 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 03:02:03.515344 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.514943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58598)
0619 03:02:03.515097 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.515180 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 03:02:03.515528 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:34529: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
W20260619 03:02:03.647789 13855 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 03:02:03.648126 13855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:02:03.648257 13855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:02:03.648388 13855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:02:03.655510 13855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:02:03.655622 13855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:02:03.655752 13855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:02:03.655838 13855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:02:03.655958 13855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:02:03.658749 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } attempt: 4
I20260619 03:02:03.659812 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.659318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58602)
0619 03:02:03.659592 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.659684 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 03:02:03.660773 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.660452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58614)
0619 03:02:03.660594 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.660668 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 03:02:03.660946 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.61:34529: Network error: Client connection negotiation failed: client connection to 127.22.230.61:34529: connect: Connection refused (error 111)
I20260619 03:02:03.665815 13855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:34335,127.22.230.61:34529,127.22.230.60:36047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.61.keytab
--principal=kudu/127.22.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.61:34529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:02:03.667042 13855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:02:03.668285 13855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 03:02:03.679649 13860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:02:03.679680 13863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:02:03.679733 13861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:02:03.683951 13855 server_base.cc:1034] running on GCE node
I20260619 03:02:03.684732 13855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:02:03.686094 13855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:02:03.687323 13855 hybrid_clock.cc:648] HybridClock initialized: now 1781838123687260 us; error 38 us; skew 500 ppm
I20260619 03:02:03.687646 13855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:02:03 dist-test-slave-2gms krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:02:03.690658 13855 init.cc:377] Logged in from keytab as kudu/127.22.230.61@KRBTEST.COM (short username kudu)
I20260619 03:02:03.691946 13855 webserver.cc:460] Webserver started at http://127.22.230.61:45089/ using document root <none> and password file <none>
I20260619 03:02:03.692564 13855 fs_manager.cc:359] Metadata directory not provided
I20260619 03:02:03.692643 13855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:02:03.692890 13855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:02:03.695638 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data/instance:
uuid: "2b202193f71145d098b8ffabc923aaea"
format_stamp: "Formatted at 2026-06-19 03:02:03 on dist-test-slave-2gms"
server_key: "4ddda75133777df2e82dbbdbd39bee32"
server_key_iv: "ec8d586cddf5cf272fd44ab9251635d4"
server_key_version: "encryptionkey@0"
I20260619 03:02:03.696342 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal/instance:
uuid: "2b202193f71145d098b8ffabc923aaea"
format_stamp: "Formatted at 2026-06-19 03:02:03 on dist-test-slave-2gms"
server_key: "4ddda75133777df2e82dbbdbd39bee32"
server_key_iv: "ec8d586cddf5cf272fd44ab9251635d4"
server_key_version: "encryptionkey@0"
I20260619 03:02:03.701007 13855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 03:02:03.704330 13870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:02:03.705490 13855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 03:02:03.705621 13855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal
uuid: "2b202193f71145d098b8ffabc923aaea"
format_stamp: "Formatted at 2026-06-19 03:02:03 on dist-test-slave-2gms"
server_key: "4ddda75133777df2e82dbbdbd39bee32"
server_key_iv: "ec8d586cddf5cf272fd44ab9251635d4"
server_key_version: "encryptionkey@0"
I20260619 03:02:03.705775 13855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:02:03.731083 13855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:02:03.734635 13855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:02:03.734838 13855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:02:03.752626 13855 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.61:34529
I20260619 03:02:03.752651 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.61:34529 every 8 connection(s)
I20260619 03:02:03.753962 13855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/data/info.pb
I20260619 03:02:03.757516 13922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 03:02:03.761315 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13855
I20260619 03:02:03.761499 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-1/wal/instance
I20260619 03:02:03.761996 23448 external_mini_cluster.cc:1428] Setting key 67f78d7b195d57d8c20791f1f9b1c418
I20260619 03:02:03.765403 13922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 03:02:03 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:02:03.787670 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.768744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.22.230.62:34335)
0619 03:02:03.768971 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:03.768977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:03.770084 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 03:02:03.771913 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:03.771922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:03.772015 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:03.772136 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:03.773496 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.774205 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.776684 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.777108 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.777207 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.781196 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:03.781201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:03.781204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:03.781236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:03.784363 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.785296 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.785300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.785303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.785417 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.785874 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.785877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.785879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.786137 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:03.786244 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:03.787173 (+ 929us) server_negotiation.cc:299] Negotiation successful
0619 03:02:03.787418 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 03:02:03.789141 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.769139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34335 (local address 127.0.0.1:49956)
0619 03:02:03.769909 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.769960 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:03.771646 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:03.772316 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:03.772323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:03.772816 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:03.773317 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.773334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.774378 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.774386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.776530 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.776547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.777340 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.777349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.777686 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.778394 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:03.778426 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:03.780976 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:03.784507 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.784518 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.784535 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.785170 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.785520 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.785527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.785534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.785762 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.786268 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:03.786278 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:03.786788 (+ 510us) client_negotiation.cc:769] Sending connection context
0619 03:02:03.787078 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 03:02:03.787619 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":324,"threads_started":1}
I20260619 03:02:03.790963 13922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.230.60@KRBTEST.COM" created.
Jun 19 03:02:03 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:02:03.809684 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.791760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58630)
0619 03:02:03.791965 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.791984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:03.792085 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:03.792994 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:03.792997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:03.793397 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:03.793529 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.793537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.794911 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.794919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.796280 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.796299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.797111 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.797123 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.797340 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.797905 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:03.797930 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:03.800273 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:03.807168 (+ 6895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.807174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.807178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.807520 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.807895 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.807899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.807903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.808039 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.808686 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:03.808693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:03.808841 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 03:02:03.809026 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 03:02:03.809292 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 03:02:03.809916 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.791958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.22.230.61:34529)
0619 03:02:03.792611 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:03.792617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:03.792643 (+ 26us) server_negotiation.cc:407] Connection header received
0619 03:02:03.792683 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:03.792689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:03.792850 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:03.792958 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:03.793675 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.794782 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.796448 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.796955 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.797050 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.800471 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:03.800481 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:03.800491 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:03.800521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:03.807031 (+ 6510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.807643 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.807650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.807654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.807766 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.808147 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.808154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.808157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.808577 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:03.808786 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:03.809169 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 03:02:03.809418 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":509,"thread_start_us":368,"threads_started":1}
I20260619 03:02:03.812256 13922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:02:03.813431 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.813006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35046)
0619 03:02:03.813176 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.813293 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 03:02:03.813673 13872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:03.815207 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.814897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35050)
0619 03:02:03.815035 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.815114 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 03:02:03.815426 13922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab.
Entry for principal kudu/127.22.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab.
I20260619 03:02:03.827780 23448 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
/tmp/dist-test-taskrIFYxA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:36047
--webserver_interface=127.22.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab
--principal=kudu/127.22.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.230.20:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.230.62:34335,127.22.230.61:34529,127.22.230.60:36047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 03:02:03.864781 13922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 1
I20260619 03:02:03.865973 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.865440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35062)
0619 03:02:03.865662 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.865791 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 03:02:03.867051 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.866676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35076)
0619 03:02:03.866836 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.866926 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 03:02:03.867254 13922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:03.935499 13922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 2
I20260619 03:02:03.936813 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.936239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35088)
0619 03:02:03.936496 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.936620 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 03:02:03.937817 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.937455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35102)
0619 03:02:03.937600 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.937681 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 03:02:03.938036 13922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:03.960187 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } attempt: 5
Jun 19 03:02:03 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:02:03.972949 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.961003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.22.230.61:34529)
0619 03:02:03.961190 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:03.961195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:03.961216 (+ 21us) server_negotiation.cc:407] Connection header received
0619 03:02:03.961427 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:03.961432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:03.961494 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:03.961601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:03.962365 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.963115 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.964258 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.964680 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.964813 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.967812 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:03.967817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:03.967820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:03.967851 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:03.970401 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.971094 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.971100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.971106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.971227 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:03.971594 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:03.971600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:03.971603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:03.971883 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:03.972034 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:03.972409 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 03:02:03.972615 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 03:02:03.972944 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.960898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58642)
0619 03:02:03.961105 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.961125 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:03.961256 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:03.961635 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:03.961639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:03.962031 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:03.962209 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.962217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.963245 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.963254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.964101 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:03.964112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:03.964833 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:03.964838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:03.965064 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:03.965679 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:03.965696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:03.967629 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:03.970535 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.970541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.970546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.970945 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.971337 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:03.971342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:03.971346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:03.971488 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:03.971998 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:03.972003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:03.972163 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 03:02:03.972302 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 03:02:03.972589 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 03:02:03.974382 13848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:02:03.975353 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.975002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35118)
0619 03:02:03.975155 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.975228 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 03:02:03.976261 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:03.975965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35132)
0619 03:02:03.976099 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:03.976176 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 03:02:03.976438 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:04.027661 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 1
I20260619 03:02:04.028756 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.028275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35148)
0619 03:02:04.028510 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.028594 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 03:02:04.029238 13922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 3
I20260619 03:02:04.029893 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.029528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35156)
0619 03:02:04.029696 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.029754 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 03:02:04.030150 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:04.030246 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.029788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35172)
0619 03:02:04.029985 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.030075 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 03:02:04.031304 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.030913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35180)
0619 03:02:04.031140 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.031204 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260619 03:02:04.031500 13922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:04.098379 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 2
I20260619 03:02:04.099561 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.099113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35188)
0619 03:02:04.099317 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.099415 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 03:02:04.100479 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.100173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35192)
0619 03:02:04.100313 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.100386 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 03:02:04.100668 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
W20260619 03:02:04.154935 13929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 03:02:04.155278 13929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 03:02:04.155401 13929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 03:02:04.155534 13929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 03:02:04.162514 13929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 03:02:04.162658 13929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 03:02:04.162783 13929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 03:02:04.162873 13929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 03:02:04.162979 13929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 03:02:04.172185 13929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.230.20:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.230.62:34335,127.22.230.61:34529,127.22.230.60:36047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/krb5kdc/kudu_127.22.230.60.keytab
--principal=kudu/127.22.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.230.60:36047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 01:54:22 UTC on ccycloud.quasar-vafwje.root.comops.site
build id 4454977
ASAN enabled
I20260619 03:02:04.173378 13929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 03:02:04.174662 13929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 03:02:04.174634 13922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 4
I20260619 03:02:04.175697 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.175231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35198)
0619 03:02:04.175468 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.175548 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 03:02:04.176645 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.176343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35206)
0619 03:02:04.176493 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.176551 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 03:02:04.176823 13922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
W20260619 03:02:04.186393 13935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:02:04.186311 13934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 03:02:04.186362 13937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 03:02:04.187067 13929 server_base.cc:1034] running on GCE node
I20260619 03:02:04.187680 13929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 03:02:04.188912 13929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 03:02:04.190119 13929 hybrid_clock.cc:648] HybridClock initialized: now 1781838124190068 us; error 22 us; skew 500 ppm
I20260619 03:02:04.190377 13929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838124, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 03:02:04.193497 13929 init.cc:377] Logged in from keytab as kudu/127.22.230.60@KRBTEST.COM (short username kudu)
I20260619 03:02:04.194885 13929 webserver.cc:460] Webserver started at http://127.22.230.60:44237/ using document root <none> and password file <none>
I20260619 03:02:04.195520 13929 fs_manager.cc:359] Metadata directory not provided
I20260619 03:02:04.195626 13929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 03:02:04.195940 13929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 03:02:04.198704 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data/instance:
uuid: "596ecfc04541402f9af3bd9b09600b20"
format_stamp: "Formatted at 2026-06-19 03:02:04 on dist-test-slave-2gms"
server_key: "d49a5f3d20670f5db3001151a95dffbf"
server_key_iv: "5f17a7cfc49636fbfad44930995c8c33"
server_key_version: "encryptionkey@0"
I20260619 03:02:04.199473 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal/instance:
uuid: "596ecfc04541402f9af3bd9b09600b20"
format_stamp: "Formatted at 2026-06-19 03:02:04 on dist-test-slave-2gms"
server_key: "d49a5f3d20670f5db3001151a95dffbf"
server_key_iv: "5f17a7cfc49636fbfad44930995c8c33"
server_key_version: "encryptionkey@0"
I20260619 03:02:04.204293 13929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 03:02:04.206867 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 3
I20260619 03:02:04.207895 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.207383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35220)
0619 03:02:04.207642 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.207735 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 03:02:04.207894 13944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 03:02:04.208942 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.208525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35232)
0619 03:02:04.208713 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.208784 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 03:02:04.209177 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.230.60:36047: Network error: Client connection negotiation failed: client connection to 127.22.230.60:36047: connect: Connection refused (error 111)
I20260619 03:02:04.209754 13929 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 03:02:04.209926 13929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data,/tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal
uuid: "596ecfc04541402f9af3bd9b09600b20"
format_stamp: "Formatted at 2026-06-19 03:02:04 on dist-test-slave-2gms"
server_key: "d49a5f3d20670f5db3001151a95dffbf"
server_key_iv: "5f17a7cfc49636fbfad44930995c8c33"
server_key_version: "encryptionkey@0"
I20260619 03:02:04.210179 13929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 03:02:04.230281 13929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 03:02:04.233963 13929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 03:02:04.234216 13929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 03:02:04.251189 13929 rpc_server.cc:307] RPC server started. Bound to: 127.22.230.60:36047
I20260619 03:02:04.251209 13995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.230.60:36047 every 8 connection(s)
I20260619 03:02:04.252413 13929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/data/info.pb
I20260619 03:02:04.253376 23448 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu as pid 13929
I20260619 03:02:04.253558 23448 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrIFYxA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781837774831470-23448-0/minicluster-data/master-2/wal/instance
I20260619 03:02:04.254114 23448 external_mini_cluster.cc:1428] Setting key feb075170a4d2577992a3b7b8377d595
I20260619 03:02:04.256906 13996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:02:04.266027 13996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } has no permanent_uuid. Determining permanent_uuid...
I20260619 03:02:04.274968 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.255808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.22.230.60:36047)
0619 03:02:04.256611 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.256622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.256660 (+ 38us) server_negotiation.cc:407] Connection header received
0619 03:02:04.258321 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.258342 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.258766 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.259133 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.260479 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.261704 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.262985 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.263648 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.263834 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.267694 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.267705 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.267719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.267757 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.271404 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.272017 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.272024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.272032 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.272208 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.272549 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.272553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.272555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.273029 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.273241 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.273627 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.273949 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":340,"threads_started":1}
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838124, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
I20260619 03:02:04.284193 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.268869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.22.230.62:34335)
0619 03:02:04.269084 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.269090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.269852 (+ 762us) server_negotiation.cc:407] Connection header received
0619 03:02:04.270129 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.270136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.270217 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.270343 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.271529 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.272397 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.274042 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.274471 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.274616 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.278145 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.278154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.278161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.278210 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.281597 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.282233 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.282239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.282242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.282384 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.282789 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.282795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.282798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.283036 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.283207 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.283674 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.283857 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 03:02:04.284221 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.268940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.62:34335 (local address 127.0.0.1:49958)
0619 03:02:04.269730 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.269768 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:04.269974 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:04.270387 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:04.270391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:04.271110 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:04.271383 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.271399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.272542 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.272552 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.273903 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.273916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.274684 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.274693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.274855 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.275496 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:04.275525 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:04.277943 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:04.281729 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.281742 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.281748 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.282112 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.282489 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.282495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.282498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.282688 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.283184 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:04.283191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:04.283370 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 03:02:04.283556 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 03:02:04.283823 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":385,"threads_started":1}
I20260619 03:02:04.285523 13996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.62@KRBTEST.COM
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838124, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
I20260619 03:02:04.293171 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.277894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.22.230.62:34335)
0619 03:02:04.279035 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.279042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.279067 (+ 25us) server_negotiation.cc:407] Connection header received
0619 03:02:04.279113 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.279119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.279198 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.279306 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.280843 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.281698 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.282940 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.283371 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.283514 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.287816 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.287821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.287824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.287853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.290471 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.291099 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.291104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.291108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.291230 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.291651 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.291655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.291657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.291904 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.292141 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.292593 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.292783 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"spinlock_wait_cycles":1408,"thread_start_us":827,"threads_started":1}
I20260619 03:02:04.298391 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.286304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.61:34529 (local address 127.0.0.1:58658)
0619 03:02:04.286459 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.286476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:04.286628 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:04.286966 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:04.286970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:04.287305 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:04.287424 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.287431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.288335 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.288343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.289198 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.289208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.289895 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.289901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.290074 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.290556 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:04.290614 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:04.292466 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:04.296147 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.296151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.296153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.296479 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.296868 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.296871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.296873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.296991 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.297518 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:04.297522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:04.297667 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 03:02:04.297804 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 03:02:04.298057 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 03:02:04.298475 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.286228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.22.230.61:34529)
0619 03:02:04.286433 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.286439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.286558 (+ 119us) server_negotiation.cc:407] Connection header received
0619 03:02:04.286784 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.286788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.286848 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.286949 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.287561 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.288207 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.289346 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.289747 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.289876 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.292666 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.292673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.292679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.292718 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.296030 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.296593 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.296598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.296604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.296761 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.297098 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.297108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.297111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.297397 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.297552 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.297930 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.298135 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 03:02:04.299340 13996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.230.61@KRBTEST.COM
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838124, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.60@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:02:04.309875 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.294936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.22.230.61:34529)
0619 03:02:04.295508 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.295512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.295532 (+ 20us) server_negotiation.cc:407] Connection header received
0619 03:02:04.295585 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.295588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.295637 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.295716 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.297146 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.297978 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.299526 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.300018 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.300161 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.304288 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.304295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.304300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.304331 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.307094 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.307868 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.307872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.307874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.307982 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.308412 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.308418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.308425 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.308660 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.308849 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.309251 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.309482 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":281,"threads_started":1}
I20260619 03:02:04.312122 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.300145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.22.230.60:36047)
0619 03:02:04.300284 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.300289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.300302 (+ 13us) server_negotiation.cc:407] Connection header received
0619 03:02:04.300343 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.300346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.300390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.300482 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.301077 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.301717 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.302872 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.303440 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.303584 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.306755 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.306761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.306765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.306803 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.309674 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.310351 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.310356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.310359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.310493 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.310852 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.310859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.310862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.311149 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.311291 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.311623 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.311816 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":48}
I20260619 03:02:04.312431 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.299933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35256)
0619 03:02:04.300074 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.300094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:04.300178 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:04.300497 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:04.300501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:04.300846 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:04.300958 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.300965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.301829 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.301837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.302732 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.302742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.303599 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.303605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.303781 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.304302 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:04.304322 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:04.306541 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:04.309805 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.309809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.309812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.310235 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.310612 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.310616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.310619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.310746 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.311249 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:04.311254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:04.311396 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 03:02:04.311553 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 03:02:04.312009 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":126}
I20260619 03:02:04.314649 23448 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 03:02:04.317158 13996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: Bootstrap starting.
I20260619 03:02:04.316787 13823 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49974
I20260619 03:02:04.320210 13996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: Neither blocks nor log segments found. Creating new log.
I20260619 03:02:04.321178 13996 log.cc:824] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: Log is configured to *not* fsync() on all Append() calls
I20260619 03:02:04.324081 13996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: No bootstrap required, opened a new log
I20260619 03:02:04.330685 13996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.330984 13996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596ecfc04541402f9af3bd9b09600b20, State: Initialized, Role: FOLLOWER
I20260619 03:02:04.331384 13996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.332237 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.332638 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.333179 13996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 03:02:04.342080 14015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 03:02:04.342197 14015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 03:02:04.339923 13929 master_runner.cc:186] Error getting master registration for 127.22.230.62:34335: OK, instance_id { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" instance_seqno: 1781838123251769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 03:02:04.350719 13929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 03:02:04.386379 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 4
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.62@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:02:04.402679 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.387069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35264)
0619 03:02:04.387246 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.387273 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:04.387369 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:04.387737 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:04.387742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:04.388260 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:04.388469 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.388481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.389886 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.389891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.391443 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.391459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.392538 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.392543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.392739 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.393413 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:04.393434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:04.396045 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:04.400009 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.400014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.400018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.400505 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.400895 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.400901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.400905 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.401066 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.401707 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:04.401715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:04.401875 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 03:02:04.402023 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 03:02:04.402284 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 03:02:04.402696 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.387111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.22.230.60:36047)
0619 03:02:04.387301 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.387307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.387330 (+ 23us) server_negotiation.cc:407] Connection header received
0619 03:02:04.387487 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.387492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.387574 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.387678 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.388631 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.389754 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.391595 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.392380 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.392516 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.396212 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.396222 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.396227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.396262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.399875 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.400628 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.400634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.400638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.400783 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.401176 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.401181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.401185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.401590 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.401759 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.402142 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.402341 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 03:02:04.408336 13848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b: Bootstrap starting.
I20260619 03:02:04.412187 13848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b: Neither blocks nor log segments found. Creating new log.
I20260619 03:02:04.413422 13848 log.cc:824] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b: Log is configured to *not* fsync() on all Append() calls
I20260619 03:02:04.416292 13848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b: No bootstrap required, opened a new log
I20260619 03:02:04.421193 13848 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.421504 13848 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca877a1ecd744dd9d5914a0aa3c7d4b, State: Initialized, Role: FOLLOWER
I20260619 03:02:04.421958 13848 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.422896 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.423262 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.424090 13848 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:02:04.424188 13823 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 03:02:04.424384 13823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:02:04.428479 13823 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.430815 13823 leader_election.cc:288] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [CANDIDATE]: Term 1 election: Requested vote from peers 2b202193f71145d098b8ffabc923aaea (127.22.230.61:34529), 596ecfc04541402f9af3bd9b09600b20 (127.22.230.60:36047)
I20260619 03:02:04.431389 13971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "596ecfc04541402f9af3bd9b09600b20"
I20260619 03:02:04.431843 13971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:02:04.435444 13971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca877a1ecd744dd9d5914a0aa3c7d4b in term 1.
I20260619 03:02:04.436120 13798 leader_election.cc:302] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596ecfc04541402f9af3bd9b09600b20, fca877a1ecd744dd9d5914a0aa3c7d4b; no voters:
I20260619 03:02:04.436623 14018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 03:02:04.437327 14018 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [term 1 LEADER]: Becoming Leader. State: Replica: fca877a1ecd744dd9d5914a0aa3c7d4b, State: Running, Role: LEADER
I20260619 03:02:04.437944 14018 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.441766 14025 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca877a1ecd744dd9d5914a0aa3c7d4b. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.442121 14025 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: This master's current role is: LEADER
I20260619 03:02:04.443493 14028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 03:02:04.449744 14028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 03:02:04.476006 13922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } attempt: 5
W20260619 03:02:04.486054 13950 tablet.cc:2369] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 03:02:04.490887 13802 tablet.cc:2369] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 03:02:04.498154 13971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [term 1 FOLLOWER]: Refusing update from remote peer fca877a1ecd744dd9d5914a0aa3c7d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:02:04.500386 14025 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 19 03:02:04 dist-test-slave-2gms krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781838123, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.230.61@KRBTEST.COM for kudu/127.22.230.60@KRBTEST.COM
I20260619 03:02:04.528637 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.464371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.22.230.62:34335)
0619 03:02:04.464609 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.464618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.476541 (+ 11923us) server_negotiation.cc:407] Connection header received
0619 03:02:04.476842 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.476851 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.476951 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.477036 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.491596 (+ 14560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.492688 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.494472 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.495207 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.495343 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.506374 (+ 11031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.506384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.506394 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.506434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.510485 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.520598 (+ 10113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.520605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.520611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.520804 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.524024 (+ 3220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.524032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.524042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.524481 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.524762 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.527980 (+ 3218us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.528231 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 03:02:04.518781 13781 master_runner.cc:186] Error getting master registration for 127.22.230.61:34529: OK, instance_id { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" instance_seqno: 1781838123747578 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 03:02:04.529294 13781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 03:02:04.531489 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.470541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.22.230.61:34529)
0619 03:02:04.470791 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.470800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.480404 (+ 9604us) server_negotiation.cc:407] Connection header received
0619 03:02:04.480702 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.480711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.480820 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.480906 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.491644 (+ 10738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.492688 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.496126 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.496740 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.496855 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.511442 (+ 14587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.511450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.511454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.511489 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.515652 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.521273 (+ 5621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.521279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.521283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.521438 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.525112 (+ 3674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.525118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.525122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.525501 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.525744 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.530854 (+ 5110us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.531110 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 03:02:04.535331 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.463135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.22.230.60:36047)
0619 03:02:04.463365 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.463375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.472648 (+ 9273us) server_negotiation.cc:407] Connection header received
0619 03:02:04.473013 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.473021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.473136 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.473233 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.487701 (+ 14468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.488933 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.497983 (+ 9050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.498692 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.498814 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.516600 (+ 17786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.516606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.516611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.516655 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.521980 (+ 5325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.523482 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.523487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.523491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.523642 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.526692 (+ 3050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.526700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.526706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.527098 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.527374 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.533380 (+ 6006us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.534882 (+ 1502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1194,"server-negotiator.queue_time_us":72}
I20260619 03:02:04.536105 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca877a1ecd744dd9d5914a0aa3c7d4b. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.536378 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.476593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.230.60:36047 (local address 127.0.0.1:35282)
0619 03:02:04.476957 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 03:02:04.476976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 03:02:04.477095 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 03:02:04.498717 (+ 21622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 03:02:04.498721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 03:02:04.499320 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 03:02:04.499600 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.499615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.502861 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.502867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.504720 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 03:02:04.504736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.506805 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.506811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 03:02:04.507025 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.507570 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 03:02:04.507602 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 03:02:04.511109 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 03:02:04.528022 (+ 16913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.528027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.528031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.528559 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.530684 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 03:02:04.530690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 03:02:04.530696 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 03:02:04.530871 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 03:02:04.535398 (+ 4527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 03:02:04.535403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 03:02:04.535553 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 03:02:04.535693 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 03:02:04.535981 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260619 03:02:04.536396 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.544301 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.477318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.22.230.60:36047)
0619 03:02:04.497360 (+ 20042us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.497366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.497387 (+ 21us) server_negotiation.cc:407] Connection header received
0619 03:02:04.497444 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.497449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.497528 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.497633 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.499833 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.502625 (+ 2792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.505027 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.506531 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.507430 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.511974 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 03:02:04.512017 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 03:02:04.512024 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 03:02:04.512108 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0619 03:02:04.527679 (+ 15571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.529431 (+ 1752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.529439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.529443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.530258 (+ 815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 03:02:04.531262 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 03:02:04.531269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 03:02:04.531281 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 03:02:04.534690 (+ 3409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 03:02:04.537245 (+ 2555us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.537388 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.537579 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19897,"thread_start_us":19651,"threads_started":1}
I20260619 03:02:04.551532 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: Bootstrap starting.
I20260619 03:02:04.553777 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.554023 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.556164 13922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: Neither blocks nor log segments found. Creating new log.
I20260619 03:02:04.557900 13922 log.cc:824] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: Log is configured to *not* fsync() on all Append() calls
I20260619 03:02:04.562917 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: No bootstrap required, opened a new log
I20260619 03:02:04.569567 14028 catalog_manager.cc:1347] Generated new cluster ID: b78b1d54a367437b869356b96fc9a4e1
I20260619 03:02:04.569677 14028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 03:02:04.571228 14025 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.571491 14025 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: This master's current role is: LEADER
I20260619 03:02:04.584519 13922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.585057 13922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b202193f71145d098b8ffabc923aaea, State: Initialized, Role: FOLLOWER
I20260619 03:02:04.585568 13922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } }
I20260619 03:02:04.596573 13897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2b202193f71145d098b8ffabc923aaea"
I20260619 03:02:04.596556 13922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 03:02:04.601058 13896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [term 0 FOLLOWER]: Advancing to term 1
I20260619 03:02:04.600963 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.601400 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.605782 13896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [term 1 FOLLOWER]: Refusing update from remote peer fca877a1ecd744dd9d5914a0aa3c7d4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 03:02:04.611202 13897 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca877a1ecd744dd9d5914a0aa3c7d4b in term 1.
I20260619 03:02:04.612901 14018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 03:02:04.650666 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca877a1ecd744dd9d5914a0aa3c7d4b. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.651012 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.650967 14056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.651311 14056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea [sys.catalog]: This master's current role is: FOLLOWER
I20260619 03:02:04.684458 14018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca877a1ecd744dd9d5914a0aa3c7d4b" member_type: VOTER last_known_addr { host: "127.22.230.62" port: 34335 } } peers { permanent_uuid: "2b202193f71145d098b8ffabc923aaea" member_type: VOTER last_known_addr { host: "127.22.230.61" port: 34529 } } peers { permanent_uuid: "596ecfc04541402f9af3bd9b09600b20" member_type: VOTER last_known_addr { host: "127.22.230.60" port: 36047 } } }
I20260619 03:02:04.684791 14018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b [sys.catalog]: This master's current role is: LEADER
I20260619 03:02:04.693632 14028 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 03:02:04.695869 14028 catalog_manager.cc:1504] Loading token signing keys...
I20260619 03:02:04.716279 14063 catalog_manager.cc:1259] Loaded cluster ID: b78b1d54a367437b869356b96fc9a4e1
I20260619 03:02:04.716367 14063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: loading cluster ID for follower catalog manager: success
I20260619 03:02:04.719758 14063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: acquiring CA information for follower catalog manager: success
I20260619 03:02:04.722064 14063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b202193f71145d098b8ffabc923aaea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 03:02:04.729094 14028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca877a1ecd744dd9d5914a0aa3c7d4b: Generated new TSK 0
I20260619 03:02:04.730029 14028 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 03:02:04.864086 14001 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:04.843476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50002 (local address 127.22.230.62:34335)
0619 03:02:04.843681 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.843688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.861020 (+ 17332us) server_negotiation.cc:407] Connection header received
0619 03:02:04.862496 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.862501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.862626 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.862713 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.863816 (+ 1103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50002: BlockingRecv error: recv got EOF from 127.0.0.1:50002 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 03:02:04.865583 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:04.845368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.22.230.60:36047)
0619 03:02:04.845575 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.845588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.861958 (+ 16370us) server_negotiation.cc:407] Connection header received
0619 03:02:04.862165 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.862172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.862281 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.862393 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.865363 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35288: BlockingRecv error: recv got EOF from 127.0.0.1:35288 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 03:02:04.867437 14002 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:04.859211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.22.230.61:34529)
0619 03:02:04.859388 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.859394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.865661 (+ 6267us) server_negotiation.cc:407] Connection header received
0619 03:02:04.866086 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.866092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.866194 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.866265 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.867180 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58686: BlockingRecv error: recv got EOF from 127.0.0.1:58686 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 03:02:04.911582 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.896504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.22.230.61:34529)
0619 03:02:04.896714 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.896722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.903460 (+ 6738us) server_negotiation.cc:407] Connection header received
0619 03:02:04.903958 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.903967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.904187 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.904320 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:04.905008 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.906209 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.908363 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.909079 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.909173 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.910041 (+ 868us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:04.910769 (+ 728us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:04.910891 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.911136 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.911225 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 03:02:04.924453 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:04.918760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.22.230.60:36047)
0619 03:02:04.918990 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.918999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.922803 (+ 3804us) server_negotiation.cc:407] Connection header received
0619 03:02:04.923024 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.923032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.923156 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.923273 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.924195 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35292: BlockingRecv error: recv got EOF from 127.0.0.1:35292 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 03:02:04.935371 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.928441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.22.230.61:34529)
0619 03:02:04.928599 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.928604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.929350 (+ 746us) server_negotiation.cc:407] Connection header received
0619 03:02:04.929548 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.929554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.929638 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.929737 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:04.930208 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.931369 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.933261 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.933966 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.934123 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.934613 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:04.934739 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:04.934835 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.934992 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.935063 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 03:02:04.944713 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:04.941563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.22.230.60:36047)
0619 03:02:04.941720 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.941725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.943106 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 03:02:04.943379 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.943385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.943466 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.943536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.944511 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35304: BlockingRecv error: recv got EOF from 127.0.0.1:35304 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 03:02:04.962812 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.955971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.22.230.61:34529)
0619 03:02:04.956185 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.956191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.956679 (+ 488us) server_negotiation.cc:407] Connection header received
0619 03:02:04.956924 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.956936 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.957011 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.957072 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:04.957704 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.958852 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.960820 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.961463 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.961589 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.962048 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:04.962193 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:04.962280 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.962456 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.962528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 03:02:04.971619 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:04.969383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.22.230.60:36047)
0619 03:02:04.969554 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.969560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.969998 (+ 438us) server_negotiation.cc:407] Connection header received
0619 03:02:04.970251 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.970257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.970355 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.970423 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:04.971405 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35312: BlockingRecv error: recv got EOF from 127.0.0.1:35312 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 03:02:04.985630 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:04.979135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.22.230.61:34529)
0619 03:02:04.979338 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:04.979344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:04.979815 (+ 471us) server_negotiation.cc:407] Connection header received
0619 03:02:04.979975 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:04.979981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:04.980065 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:04.980162 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:04.980595 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.981745 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.983655 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:04.984308 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:04.984431 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:04.984915 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:04.985054 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:04.985141 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:04.985293 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 03:02:04.985364 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260619 03:02:05.044005 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:05.028649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.22.230.60:36047)
0619 03:02:05.028914 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.028923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.036382 (+ 7459us) server_negotiation.cc:407] Connection header received
0619 03:02:05.036593 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.036601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.036707 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.036807 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:05.043592 (+ 6785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35314: BlockingRecv error: recv got EOF from 127.0.0.1:35314 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 03:02:05.076195 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.068836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.22.230.61:34529)
0619 03:02:05.069011 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.069019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.069416 (+ 397us) server_negotiation.cc:407] Connection header received
0619 03:02:05.069636 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.069643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.069772 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.069875 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.070422 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.071559 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.073869 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.074635 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.074763 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.075331 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.075459 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.075551 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.075721 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.075812 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 03:02:05.094841 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:05.092520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.22.230.60:36047)
0619 03:02:05.092713 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.092720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.093279 (+ 559us) server_negotiation.cc:407] Connection header received
0619 03:02:05.093464 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.093470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.093599 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.093706 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:05.094628 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35328: BlockingRecv error: recv got EOF from 127.0.0.1:35328 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:02:05.122828 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.115387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.22.230.61:34529)
0619 03:02:05.115585 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.115592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.116123 (+ 531us) server_negotiation.cc:407] Connection header received
0619 03:02:05.116328 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.116333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.116414 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.116517 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.116991 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.118072 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.120635 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.121292 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.121429 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.121989 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.122150 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.122216 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.122470 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.122553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 03:02:05.136579 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:05.132079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.22.230.60:36047)
0619 03:02:05.132239 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.132247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.135107 (+ 2860us) server_negotiation.cc:407] Connection header received
0619 03:02:05.135263 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.135270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.135388 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.135501 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:05.136264 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35340: BlockingRecv error: recv got EOF from 127.0.0.1:35340 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 03:02:05.179507 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.171734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.22.230.61:34529)
0619 03:02:05.171955 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.171962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.172534 (+ 572us) server_negotiation.cc:407] Connection header received
0619 03:02:05.172759 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.172768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.172882 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.172997 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.173505 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.174970 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.176957 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.177845 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.178021 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.178522 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.178737 (+ 215us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.178847 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.179011 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.179114 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 03:02:05.188720 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:05.186322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.22.230.60:36047)
0619 03:02:05.186521 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.186529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.187145 (+ 616us) server_negotiation.cc:407] Connection header received
0619 03:02:05.187334 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.187340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.187433 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.187546 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:05.188450 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35344: BlockingRecv error: recv got EOF from 127.0.0.1:35344 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 03:02:05.263847 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.256557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.22.230.61:34529)
0619 03:02:05.256786 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.256794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.257356 (+ 562us) server_negotiation.cc:407] Connection header received
0619 03:02:05.257578 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.257584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.257667 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.257758 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.258295 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.259359 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.261794 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.262419 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.262535 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.263123 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.263248 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.263361 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.263502 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.263576 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 03:02:05.272774 14040 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 03:02:05.270759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.22.230.60:36047)
0619 03:02:05.270926 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.270932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.271343 (+ 411us) server_negotiation.cc:407] Connection header received
0619 03:02:05.271534 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.271540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.271622 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.271716 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 03:02:05.272580 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35346: BlockingRecv error: recv got EOF from 127.0.0.1:35346 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 03:02:05.343941 14015 catalog_manager.cc:1259] Loaded cluster ID: b78b1d54a367437b869356b96fc9a4e1
I20260619 03:02:05.344046 14015 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: loading cluster ID for follower catalog manager: success
I20260619 03:02:05.346279 14015 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: acquiring CA information for follower catalog manager: success
I20260619 03:02:05.347932 14015 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 596ecfc04541402f9af3bd9b09600b20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 03:02:05.409682 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.404505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.22.230.61:34529)
0619 03:02:05.404728 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.404734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.405156 (+ 422us) server_negotiation.cc:407] Connection header received
0619 03:02:05.405393 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.405397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.405461 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.405553 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.405937 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.406633 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.408034 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.408489 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.408617 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.408996 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.409089 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.409179 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.409316 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.409389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 03:02:05.419601 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.414688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.22.230.60:36047)
0619 03:02:05.414905 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.414911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.415211 (+ 300us) server_negotiation.cc:407] Connection header received
0619 03:02:05.415325 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.415329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.415457 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.415546 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.415849 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.416500 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.417640 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.418029 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.418110 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.418520 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.419043 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.419150 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.419206 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.419260 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:02:05.433445 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.424987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.22.230.60:36047)
0619 03:02:05.425160 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.425168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.427383 (+ 2215us) server_negotiation.cc:407] Connection header received
0619 03:02:05.427542 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.427548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.427631 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.427717 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.428295 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.429257 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.431570 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.432166 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.432298 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.432730 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.432862 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.432949 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.433095 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.433162 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 03:02:05.433691 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.424650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.22.230.62:34335)
0619 03:02:05.426887 (+ 2237us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.426893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.427260 (+ 367us) server_negotiation.cc:407] Connection header received
0619 03:02:05.427536 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.427541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.427754 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.427936 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.428574 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.429540 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.431216 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.431719 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.431838 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.432430 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.432740 (+ 310us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.432855 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.433297 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.433400 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2084,"thread_start_us":1975,"threads_started":1}
I20260619 03:02:05.433786 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 03:02:05.424763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.22.230.61:34529)
0619 03:02:05.424925 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 03:02:05.424930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 03:02:05.427673 (+ 2743us) server_negotiation.cc:407] Connection header received
0619 03:02:05.427937 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 03:02:05.427942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 03:02:05.428033 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 03:02:05.428099 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 03:02:05.428837 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.429589 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.431813 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 03:02:05.432203 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 03:02:05.432350 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 03:02:05.432874 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 03:02:05.432992 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 03:02:05.433068 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 03:02:05.433430 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 03:02:05.433493 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 03:02:05.437507 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13781
I20260619 03:02:05.438723 13843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:02:05.543030 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13781
I20260619 03:02:05.564320 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13855
I20260619 03:02:05.565389 13917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:02:05.668177 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13855
I20260619 03:02:05.688562 23448 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu pid 13929
I20260619 03:02:05.689656 13991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 03:02:05.790036 23448 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrIFYxA/build/asan/bin/kudu with pid 13929
2026-06-19T03:02:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4271 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8644 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (350915 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST